Virtual Private Label Cloud Console Customization

Abstract

Novel techniques are disclosed for enabling customizable consoles for various virtual private label clouds (vPLCs). In some embodiments, a single console server may run multiple consoles for multiple vPLCs and CSPs. In other embodiments, a single console server may be dedicated to vPLC-specific consoles. In some embodiments, console customization, including a set of console user interface (UI) customizations, may be performed for each vPLC-specific console.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a formal application under 35 U.S.C. §119(e) of and claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 407,571, "VIRTUAL PRIVATE LABEL CLOUD," filed September 16, 2022, the entire contents of which are incorporated herein by reference.

[0002] This application is also related to the following applications, the entire contents of each of which are incorporated herein by reference:

[0003] (1) U.S. Patent Application No. 18 / 368,881, "VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311130 (344800US)). (2) U.S. Patent Application No. 18 / 368,870, "RESOURCE ALLOCATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311140 (345000US)). (3) U.S. Patent Application No. 18 / 368,877, "ENDPOINTS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311138 (345100US)). (4) U.S. Patent Application No. 18 / 368,884, "IDENTITY MANAGEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311139 (345200US)). (5) U.S. Patent Application No. 18 / 468,024, "METADATA CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311136 (345300US)). (6) U.S. Patent Application No. 18 / 468,037, "REMOTE DATA PLANES FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311141 (345400US)). (7) U.S. Patent Application No. 18 / 468,238, "CONNECTIVITY FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311142 (345500US)). (8) U.S. Patent Application No. 18 / 468,047, "RESOURCE USAGE MONITORING, BILLING AND ENFORCEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315390 (346900US)). (9) U.S. Patent Application No. 18 / 468,044, "CLOUD INFRASTRUCTURE-BASED ONLINE PUBLISHING PLATFORMS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315384 (346800US)). Field The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for enabling customizable consoles for various virtual private label clouds (vPLCs). A vPLC can be created for a cloud service provider (CSP) reseller using a CSP-provided infrastructure in a region and used to offer one or more reseller-supplied cloud services to the reseller's customers. [Background technology]

[0004] background A cloud console provides a convenient way for users to access cloud services and resources. Customization of the console can improve performance by supporting the user's experience and providing more efficient management. However, there is a need for improved console customization in cloud computing. Summary of the Invention [Means for solving the problem]

[0005] overview The present disclosure generally relates to techniques for providing cloud services. More particularly, a novel technique is disclosed for enabling customizable consoles for various virtual private label clouds (vPLCs). A vPLC is created for a cloud service provider (CSP) reseller using a CSP-provided infrastructure in a region, allowing the reseller to offer one or more reseller-supplied cloud services to the reseller's customers.

[0006] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0007] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region.

[0008] In one embodiment, a CSP-provided regional infrastructure in a region can be used to both provide cloud services to the CSP's customers and provide reseller-supplied cloud services to the reseller's customers, who are also the CSP's customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those services. The reseller, in turn, uses the CSP-provided infrastructure to provide the reseller-branded cloud services to the reseller's customers. In this way, a CSP-supplied cloud service can be provided to customers of the CSP by using a first portion of the infrastructure that the CSP provides in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller by using a second portion of the infrastructure that the CSP provides in the region and that is assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller by using a third portion of the infrastructure that the CSP provides in the region and that is assigned to a second vPLC created for a second reseller, and so on.

[0009] In one embodiment, a technology is provided that includes a method, comprising: providing one or more CSP-supplied cloud services to one or more customers of a cloud service provider (CSP) by using a first portion of CSP-provided infrastructure in a first region; and generating a first virtual private label cloud (vPLC) for a first reseller based on the CSP-provided infrastructure, where generating the first vPLC includes allocating a second portion of the CSP-provided infrastructure to the first vPLC, the method further including: providing the one or more first reseller-supplied cloud services to one or more customers of the first reseller by using the first vPLC; providing a first console including a first set of user interfaces (UIs) to one or more customers of the CSP by using the CSP-provided infrastructure; and providing a second console including a second set of user interfaces to one or more customers of the first reseller by using the CSP-provided infrastructure, where the second console is different from the first console.

[0010] In yet another embodiment, the first console is executed by a first server and the second console is executed by a second server different from the first server.

[0011] In yet another embodiment, the first console and the second console are executed by the same server.

[0012] In yet another embodiment, the first set of UIs includes a UI customized for the CSP and the second set of UIs includes a UI customized for the first reseller.

[0013] In yet another embodiment, the first console is associated with a first set of endpoints associated with the CSP-supplied cloud service, and endpoints in the first set of endpoints are callable via the first console.

[0014] In yet another embodiment, the second console is associated with a second set of endpoints associated with the first reseller-supplied cloud service, and endpoints in the second set of endpoints are callable via the second console.

[0015] In yet another embodiment, the second console is associated with the namespace of the first vPLC.

[0016] In yet another embodiment, the method further includes generating a second vPLC for the second reseller based on the CSP-provided infrastructure, where generating the second vPLC includes allocating a third portion of the CSP-provided infrastructure to the second vPLC, and the method further includes providing one or more second reseller-supplied cloud services to one or more customers of the second reseller by using the second vPLC; and providing a third console including a third set of UIs to the one or more customers of the second reseller by using the CSP-provided infrastructure, where the first console, the second console, and the third console are different.

[0017] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0018] In various embodiments, a non-transitory computer-readable medium is provided that stores computer-executable instructions that, when executed by one or more processors of a computer system, cause the one or more processors to perform one or more methods disclosed herein.

[0019] In various embodiments, a computer program product is provided that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0020] The techniques described above and below can be implemented in many ways and in many contexts. Several exemplary implementations and contexts are provided by reference to the drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] FIG. 2 is a simplified architectural diagram of physical components in a physical network within CSPI, according to one embodiment. [Figure 3] FIG. 1 illustrates an exemplary configuration within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to an embodiment. [Figure 4] FIG. 1 illustrates a connection between a host machine and an NVD to provide I / O virtualization that supports multi-tenancy, according to one embodiment. [Figure 5] FIG. 1 is a simplified block diagram of a physical network provided by CSPI, according to one embodiment. [Figure 6]1 is a block diagram illustrating an exemplary high-level architecture of a virtual private label cloud (vPLC) hosted in a data center, according to one embodiment. [Figure 7] FIG. 1 illustrates two levels of tenancy in a vPLC architecture, according to an embodiment. [Figure 8] FIG. 1 illustrates two levels of tenancy in a vPLC architecture, according to an embodiment. [Figure 9A] FIG. 1 illustrates a mechanism for storing information in a customer tenancy for a vPLC, according to an embodiment. [Figure 9B] FIG. 1 illustrates a mechanism for storing information in a customer tenancy for a vPLC, according to an embodiment. [Figure 10] 10 is a flowchart illustrating a vPLC configuration process according to an embodiment. [Figure 11] FIG. 1 is a block diagram illustrating an exemplary high-level architecture of multiple vPLCs hosted in a data center, according to an embodiment. [Figure 12] FIG. 1 is a block diagram illustrating a vPLC segmentation model, according to an embodiment. [Figure 13] FIG. 1 is a block diagram illustrating a vPLC nesting model, according to an embodiment. [Figure 14] FIG. 1 is a flow diagram illustrating sign-up for a reseller's customer, according to one embodiment. [Figure 15] FIG. 10 is a flow diagram illustrating a login process for a reseller's customers, according to one embodiment. [Figure 16] FIG. 1 is a flow diagram illustrating a vPLC resource allocation and provisioning process according to an embodiment. [Figure 17] FIG. 1 is a flow diagram illustrating a vPLC resource allocation and provisioning process according to an embodiment. [Figure 18] FIG. 1 is a simplified diagram illustrating a vPLC hybrid model for inter-realm communication, according to an embodiment. [Figure 19]1 is a flowchart illustrating an example process for creating a vPLC virtual region for inter-realm communication, according to an embodiment. [Figure 20] FIG. 1 illustrates a dedicated vPLC console model, according to an embodiment. [Figure 21] FIG. 1 illustrates a shared vPLC console model, according to an embodiment. [Figure 22] FIG. 1 illustrates a hybrid vPLC console model, according to an embodiment. [Figure 23] 10 is a flowchart illustrating a process for configuring vPLC console customization, according to an embodiment. [Figure 24] 10 is a flowchart illustrating a login process for a user associated with a customer of a reseller associated with a vPLC using a vPLC-specific console, according to one embodiment. [Figure 25] 1 is a flowchart illustrating a method for serving a console GUI to a user associated with a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 26] FIG. 1 is a block diagram illustrating a pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 27] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 28] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 29] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 30] FIG. 1 is a block diagram illustrating an exemplary computer system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and this description are not intended to be limiting. Use of the word "exemplary" herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0023]

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed that enable cloud infrastructure used to provide cloud service provider (CSP)-supplied cloud services to CSP customers in a region provided by the CSP and to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs"). Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code, or instructions that can be executed by one or more processors, and the like.

[0024] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0025] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region. In some embodiments, a virtual private label cloud (vPLC) represents a model and architecture that enables and facilitates CSP-provided, reseller-supplied, and reseller-branded cloud services within a CSP-provided infrastructure in a region.

[0026] Just as a CSP-provided infrastructure in a region provides cloud resources that can be accessed by the CSP's customers, a vPLC created for a reseller provides a set of resources that can be accessed by the reseller's customers. From the perspective of the reseller's customers, a vPLC is like a reseller-provided infrastructure in a region, providing resources that the reseller's customers can access. A vPLC is like a data center that delivers reseller-provided cloud services to the reseller's customers in the region that the reseller provides.

[0027] In one embodiment, a CSP-provided regional infrastructure in a region can be used to both provide cloud services to the CSP's customers and provide reseller-supplied cloud services to the reseller's customers, who are also the CSP's customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide the cloud services. Instead, the reseller uses the CSP-provided infrastructure to provide reseller-branded cloud services to the reseller's customers.

[0028] In a reseller use case, a vPLC is created for the reseller by using cloud infrastructure in a region provided by a cloud service provider (CSP). The vPLC created for the reseller can be used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In one embodiment, the CSP-provided regional infrastructure providing the services used by the reseller to provide one or more reseller-supplied services is partitioned into securely isolated portions, and these isolated portions (or partitions) are assigned to different resellers, such that the partitions assigned to one reseller are isolated from other resellers. In this manner, a CSP-supplied cloud service can be provided to customers of the CSP using a first portion of the infrastructure provided by the CSP in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller using a second portion of the infrastructure provided by the CSP in the region and assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller using a third portion of the infrastructure provided by the CSP in the region and assigned to a second vPLC created for a second reseller, etc. Each portion of the infrastructure assigned to a reseller can also be further divided into second-level, securely isolated portions to provide multi-tenant cloud infrastructure services to customers of the resellers.

[0029] A reseller can be an entity such as a company, a business, or an individual. By using the technology described herein, a reseller entity can quickly become a cloud service provider while avoiding all of the barriers to entry discussed in the background section of this disclosure. For example, a reseller does not need to acquire the infrastructure used to provide cloud services because the provided cloud services use infrastructure provided by a CSP. To the reseller's customers, it appears that the reseller is providing reseller-supplied cloud services. Thus, from the perspective of the reseller's customers, the reseller is the cloud service provider whose services the customer subscribes to. The reseller's customers may not even know or recognize the CSP whose infrastructure the reseller uses to provide reseller-supplied and reseller-branded cloud services.

[0030] The ability to provide a vPLC is offered as a service by a CSP, and one or more customers can sign up. For example, an entity that wants to provide cloud services to its customers but does not want to invest in acquiring the infrastructure required to deliver the services can sign up for this CSP-supplied vPLC service. When an entity registers for a vPLC service, it becomes a customer of the CSP, but it is a "special" customer because the CSP-provided infrastructure allocated to the entity in the form of a vPLC is used to provide the entity-supplied and branded services to the entity's customers. Thus, the customer is both a customer of the CSP and a reseller of cloud services that use the vPLC generated for the entity. For this reason, the entity is referred to as a "reseller" to distinguish it from the CSP's actual direct customers. A CSP's direct customers are entities that sign up for and consume one or more CSP-supplied and branded services, but do not use the CSP infrastructure to sell cloud services to their customers. For purposes of this disclosure, a CSP's direct customers are also referred to as "non-reseller customers" to distinguish them from resellers. With respect to the reseller, as part of the vPLC service, the CSP will provide infrastructure to the reseller in the form of vPLC, which the reseller will use to provide the reseller-branded cloud services to the reseller's customers.

[0031] A vPLC can also be used for a variety of other purposes that do not involve a reseller. As one example, a vPLC can be created and associated with a realm that is different from the realm associated with the CSP-provided regional infrastructure used to create the vPLC. For example, the CSP-provided regional infrastructure may be associated with a first realm, and the vPLC may be associated with a second realm that is different from the first realm. In such a scenario, the infrastructure corresponding to the vPLC is hosted by the first realm but is considered to belong to the second realm. This can be used for a variety of different purposes. For example, because the vPLC is associated with the second realm, it can communicate with other infrastructure in the second realm (e.g., other data centers). Thus, data centers in the second realm can communicate with the vPLC because they reside in the same realm and share the trust and identity profile of the second realm.

[0032] The creation and management of a vPLC is performed using infrastructure and services provided by a CSP, a highly complex technical task. When a vPLC is created for a reseller, technical functionality must be provided that enables the reseller to use the vPLC to provide reseller-supplied cloud services to the reseller's customers using the vPLC. This includes segregating traffic between resellers, segregating traffic of the reseller's different customers, dynamically managing the allocation of CSP-provided resources to vPLCs, managing the allocation of resources allocated to vPLCs among the vPLC's different customers, metering the usage of vPLCs allocated to the reseller and performing related billing functions, metering the usage of vPLCs allocated among the vPLC's different customers and performing related billing functions, enabling a marketplace for different resellers, and performing identity management functions for resellers and their respective customers. This disclosure describes various embodiments illustrating various architectures and corresponding methods for implementing and enabling vPLC-related functionality.

[0033] A reseller may use the vPLC created for the reseller to provide reseller-supplied and reseller-branded cloud services that provide specialized services in the domain in which the reseller specializes. The reseller-supplied services may be tailored to various customer segments. In some embodiments, the reseller-supplied cloud services may be different from the CSP-supplied cloud services. In some other embodiments, the reseller-supplied cloud services may be based on the CSP-supplied cloud services. For example, the reseller-supplied services may be customized versions of the CSP-supplied cloud services.

[0034] For example, in a first region, a first vPLC may be created from the CSP-provided infrastructure for a first reseller that is specialized in providing financial services. The first reseller's customers may be financial institutions, such as banks. The first reseller may then use the first vPLC to provide a customized, first reseller-branded financial cloud service to its customers. In the first region, a second vPLC may be created from the same CSP-provided infrastructure for a second reseller that is specialized in providing telecommunications services. The second reseller's customers may be users of telecommunications services. The second reseller may then use the second vPLC to provide a customized, second reseller-branded telecommunications cloud service to its customers. In addition to the first and second vPLCs, the CSP may use the CSP-provided infrastructure in the first region to provide CSP-supplied and branded cloud services to one or more of the CSP's direct (non-reseller) customers. In this manner, use of the same CSP-provided infrastructure in a region provides CSP-supplied and trademarked cloud services to one or more direct (non-reseller) customers of the CSP, provides customized first reseller-branded financial cloud services to customers of the first reseller, and provides customized second reseller-branded telecommunications cloud services to customers of the second reseller.

[0035] The present disclosure relates generally to techniques for providing cloud services. More particularly, novel techniques for enabling customizable consoles for various virtual private label clouds (vPLCs) are disclosed.

[0036] Some technologies have limitations that allow only one console per realm. For disaster recovery purposes, there may be a secondary console in this realm. End users of the CSP's customers can log in through the CSP's console and view resources specific to that end user. When an intermediary, such as a reseller, is added, the CSP must create a new, separate region for that intermediary. These approaches are uneconomical as they increase the cost of ownership and limit the user experience.

[0037] Additionally, a default console user interface (UI) for all customers can be counterproductive because each customer has different needs. For example, one customer may be a retailer, while another is an educational institution. A retail customer may desire a console that provides a more user-friendly display for navigating sales information, such as eye-catching product descriptions and images, eye-catching promotions, and an easy-to-find checkout process. On the other hand, an educational customer may prefer a console that provides a student- and faculty-friendly display, such as course information, latest announcements and events, and admissions and application information. Thus, a default console may be easy to navigate for one customer but difficult for another.

[0038] The techniques described in this disclosure may enable multiple console experiences through different consoles for different vPLCs operating in the same physical region. A CSP may provide a console, including a console UI and a console server, for each reseller to manage the reseller's vPLC resources and customize the console UI. For purposes of this application, a console may refer to a set of graphical user interfaces (GUIs) through which a user may interact with cloud services (e.g., IaaS, PaaS, and SaaS) and infrastructure, such as provisioning and managing compute instances, creating storage volumes, setting security policies, and monitoring resources. The set of console UIs may have navigation menus and dashboards with options that allow a user to select or input information. The consoles may be executed by one or more console servers.

[0039] In some embodiments, the technology described herein can create multiple consoles, each associated with one or more endpoints, which may have multiple IP addresses served by the same underlying infrastructure. For example, multiple consoles may be run by the same console server (referred to herein as a shared console model), or each console may be run by a dedicated console server located in the same region (referred to herein as a dedicated console model). In some embodiments, a hybrid model combining the shared console model and the dedicated console model may be possible. The backend console server may use each user's login information in conjunction with the console's extended vPLC-related information to determine who is accessing which console and respond with a customized console experience for each user. This allows for the customizable vPLC console model to share backend resources and reduce costs, while providing different console experiences customized by different resellers.

[0040] Additionally, the disclosed technology not only allows users associated with a reseller's customers to visualize cloud resources associated with the reseller's vPLC, but also allows the reseller to add additional aspects (e.g., look and feel) to each user's experience. In other words, users associated with a CSP's customers may see a default view when they log in to a standard CSP console. Users of the reseller's customers will have a different console experience when they log in through a vPLC-specific console.

[0041] Because resellers have a better understanding of each customer's needs, the flexible console customization capabilities provided by the disclosed technology enable resellers to provide better customer service, improve customer productivity, and become more competitive in the marketplace.

[0042] Figures 1-5 and the associated description provided in the "Exemplary Virtual Networking Architecture" section below describe networking concepts including network virtualization, foundation networks, overlay networks, VNICs, etc., and provide example environments in which certain embodiments described herein may be implemented. Figures 6-25 describe examples and embodiments related to the vPLC architecture and vPLC console customization described herein. Figures 26-29 illustrate example architectures for implementing a cloud infrastructure that provides one or more cloud services and may incorporate the teachings described herein. Figure 30 is a block diagram illustrating an example computer system or device according to at least one embodiment.

[0043] Exemplary Virtual Networking Architecture The term cloud service is generally used to describe services made available on demand (e.g., through a subscription model) by a cloud service provider (CSP) to users or customers through the use of systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. This allows customers to use cloud services provided by CSPs without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service.

[0044] There are multiple cloud service providers offering different types of cloud services, which come in a variety of different types or models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS).

[0045] A customer may sign up for one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer signs up or subscribes to a service offered by a CSP, a tenancy or account is created for the customer. This account allows the customer to access one or more registered cloud resources associated with the account.

[0046] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (called Cloud Service Provider Infrastructure, or CSPI) that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure.

[0047] CSPI may include interconnected high-performance compute resources, including various host machines, memory resources, and network resources, that make up a physical network, also referred to as an infrastructure network or base network. CSPI resources may be distributed across one or more data centers, which may be geographically distributed across one or more geographic regions. These physical resources may run virtualization software to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network serves as the basis for creating one or more overlay or virtual networks on the physical network. The physical network (or infrastructure network or base network) comprises physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on the physical infrastructure network. A given physical network may support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a virtual cloud network (VCN). Virtual networks are realized through the use of software virtualization technologies (e.g., hypervisors, virtualization functions realized by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that realize one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can operate over a physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer 3 networking.Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), VXLAN-IETF RFC 7348 (Virtual Extensible LAN), VPN (Virtual Private Network) (e.g., MPLS Layer-3 Virtual Private Networks (RFC 4364)), VMware's NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0048] In IaaS, a CSP-provided infrastructure (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with these infrastructure components. These services can be policy-driven, enabling IaaS users to maintain application availability and performance by implementing policies that drive load balancing. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available, distributed environment. CSPI provides high-performance compute resources and capabilities, as well as storage capacity, in a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer registers or subscribes to an IaaS service offered by a CSP, a tenancy is created for that customer, which is an isolated and secure partition within CSP where the customer may create, organize, and manage their cloud resources.

[0049] Customers can build their own virtual networks using compute, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed onto these virtual networks. For example, customers can build one or more customizable private virtual networks (referred to as virtual cloud networks (VCNs)) using resources provided by CSPI. Customers can deploy one or more customer resources, such as compute instances, into the customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. In this way, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available virtual environment. While customers do not manage or control the underlying physical resources provided by CSPI, they do control the operating system, storage, and deployed applications. Control over some networking components (e.g., firewalls) may be limited.

[0050] The CSP may provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0051] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions and safeguards are taken within CSPI to ensure that each tenant's data is isolated and not visible to other tenants.

[0052] In a physical network, a network endpoint ("endpoint") represents a computing device or system that is connected to the physical network and communicates with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These virtual network endpoints are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., an overlay MAC address) and overlay Layer 3 addresses (e.g., an overlay IP address). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) may be moved from one endpoint to another through the use of network management software. Because virtual networks are built on top of physical networks, communication between components of a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, CSPI components are configured to learn and store mappings from overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa, and use these mappings to facilitate communications and encapsulate customer traffic to facilitate routing in the virtual network.

[0053] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) of an underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity of an overlay network, such as a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in their respective VCNs without any knowledge of each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each with its own real IP address.

[0054] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include physical or underlying network components as well as virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on the physical network components. In one embodiment, the CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and may be separated by large distances, for example, across countries or continents. For example, a first region may be in Australia, another region in Japan, and yet another region in India. CSPI resources are divided among the regions so that each region has its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region typically has multiple paths connecting it to other regions in the realm.

[0055] Typically, applications are deployed to the region where they are most heavily used (i.e., deployed to the infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed to different regions for a variety of reasons, such as redundancy to mitigate the risk of region-wide events such as large-scale weather systems or earthquakes, or to meet various requirements such as jurisdictions, tax areas, and other business or societal standards.

[0056] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0057] ADs within a region are isolated from each other, fault-tolerant, and designed to minimize the likelihood of simultaneous failures. This is achieved by ADs not sharing critical infrastructure resources such as networks, physical cables, cable paths, or cable entry points. Therefore, a failure in one AD within a region is unlikely to affect the availability of other ADs in the same region. ADs within the same region can be connected to each other via low-latency, high-bandwidth networks, enabling highly available connections to other networks (e.g., the Internet, customer on-premises networks, etc.). This allows for the creation of replica systems in multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure offered by an IaaS provider grows, capacity can be added to more regions and ADs. Traffic between availability domains is typically encrypted.

[0058] In one embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A customer's tenancy or account using a CSP exists in a single realm and can be distributed across one or more regions belonging to that realm. Typically, when a customer signs up for an IaaS service, a tenancy or account for that customer is created in a customer-specific region (called the "home" region) within the realm. The customer can extend their tenancy across one or more other regions within the realm. A customer cannot access regions that are not in the realm in which their tenancy resides.

[0059] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for customers in a particular country. As yet another example, a government realm may be offered for governments, and so on. For example, a government realm may cater to a particular government and provide a higher level of security than a commercial realm. For example, OCI (Oracle Cloud Infrastructure) currently offers one realm for its commercial regions and two realms for its government cloud regions (e.g., FedRAMP and IL5 certified).

[0060] In one embodiment, an AD can be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within an AD that provides anti-affinity. Fault domains can distribute compute instances so that they do not reside on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain represents a set of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. This ensures that a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in one embodiment, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0061] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted in the cloud by CSPI is called a virtual cloud network (VCN). A customer can configure one or more virtual cloud networks (VCNs) using the allocated CSPI resources. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN may include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed in a VCN can communicate with public endpoints (“public endpoints”) accessible over a public network such as the Internet, other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), the customer's on-premises data center or network, service endpoints, and other types of endpoints.

[0062] CSPs can use CSPI to provide various services. In some cases, customers of a CSPI themselves can act as service providers and provide services using CSPI resources. Service providers may expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoints exposed by that service. These service endpoints are generally publicly accessible to users over a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are also sometimes referred to as public endpoints.

[0063] In some embodiments, a service provider may expose a service via a service endpoint (sometimes referred to as a service endpoint). Customers of the service may then access the service using the service endpoint. In some embodiments, a given service endpoint for a service may be accessible by multiple customers intending to consume the service. In other embodiments, a customer may be given a dedicated service endpoint such that only that customer may access the service using the dedicated service endpoint.

[0064] In one embodiment, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space (e.g., 10.0 / 16), which are broad private overlay IP addresses assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN exists within a single region but can span one, more, or all availability domains in the region. A gateway is a virtual interface configured for a VCN that enables traffic to communicate between the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication between different types of endpoints.

[0065] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a configuration unit or subdivision that may be created within a VCN. A VCN may have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space.

[0066] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a Layer 2 port on a switch. A VNIC is associated with a compute instance and a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet of the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. For a subnet containing a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, with each VNIC associated with one compute instance in the set of compute instances.

[0067] Each compute instance is assigned a private overlay IP address via the VNIC associated with that compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to the compute instance. In a given subnet, all VNICs use the same route table, security list, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24). For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0068] In some embodiments, a compute instance may be assigned a private overlay IP address and optionally additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet. These addresses are assigned to the same VNIC or multiple VNICs associated with the compute instance. However, each instance has a primary VNIC associated with an overlay private IP address that is created and assigned to the instance when the instance is launched (this primary VNIC cannot be deleted). Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in the same subnetwork as the primary VNIC, or in different subnetworks in the same or a different VCN.

[0069] If a compute instance resides in a public subnet, it may optionally be assigned a public IP address. A subnet can be designated as a public or private subnet at the time of creation. A private subnet means that resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs in the subnet can have public IP addresses. Customers can specify a subnet to reside in a single availability domain or multiple availability domains of a region or realm.

[0070] As described above, a VCN can be subdivided into one or more subnets. In one embodiment, a virtual router (VR) (referred to as a VCN VR or simply VR) configured for a VCN enables communication between subnets in the VCN. For a subnet within a VCN, the VR represents the subnet's logical gateway, allowing that subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are described separately below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, and a VCN VR potentially has a limited number of ports addressed by IP addresses, one port per subnet in the VCN. As such, a VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. VRs are also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address in an overlay IP address range for a subnet is reserved for a port in the VCN VR for that subnet. For example, consider two subnets in a VCN, each with an associated address range of 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with address range 10.0 / 16, an address in this range is reserved for a port in the VCN VR for that subnet. In some cases, the first IP address in this range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for a port in the VCN VR for that subnet. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port with IP address 10.1.0.1 for the second subnet.A VCN VR has a different IP address for each of the VCN's subnets.

[0071] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet through use of a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address in a range of IP addresses associated with the subnet. VNICs in the subnet can communicate with (e.g., send and receive packets from) the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the ingress / egress point for the subnet. VRs associated with a subnet within a VCN can communicate with other VRs associated with other subnets in the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions for a subnet are running on or performed by one or more NVDs that perform the VNIC functions for VNICs in the subnet.

[0072] A VCN can be configured with route tables, security rules, and DHCP options. Route tables are virtual route tables for a VCN and contain rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. A VCN's route tables can be customized to control how packets are forwarded / routed to the VCN. DHCP options represent configuration information that is automatically provided to instances at launch.

[0073] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the type of traffic (e.g., based on protocol and port) that can enter and exit instances in the VCN. Customers can choose whether a given rule is stateful or stateless. For example, a customer can allow ingress SSH traffic from anywhere to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources in that group. On the other hand, a security list contains rules that apply to all resources in any subnet that uses the security list. A VCN can be provided with a default security list that contains default security rules. DHCP options configured for a VCN provide configuration information that is automatically given to instances in the VCN at launch.

[0074] In one embodiment, configuration information for a VCN is determined and stored by a VCN control plane. The VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets and related information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs in the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, VCN state information, and other VCN-related information. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane or portions thereof to the NVD. The distributed information is stored by the NVD and may be used to update information (e.g., forwarding tables, routing tables, etc.) used to forward packets to compute instances in the VCN.

[0075] In one embodiment, VCN and subnet creation is handled by a VCN control plane (CP), and compute instance launch is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instance and then calls the VCN control plane to create and associate VNICs with the compute instance. The VCN CP also sends VCN data mappings to a VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, a VCN VP provides a distribution service responsible for providing updates to the VCN data plane. Examples of VCN control planes are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below.

[0076] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer VCN can communicate with various endpoints. These endpoints can include CSPI-hosted endpoints and endpoints external to CSPI.

[0077] Various architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, and 5 and are described below. Figure 1 is a high-level diagram of a distributed environment 100 illustrating an overlay or customer VCN hosted by CSPI, according to one embodiment. The distributed environment shown in Figure 1 includes multiple components of an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some embodiments, the distributed environment shown in Figure 1 may include more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0078] As shown in the example of FIG. 1 , distributed environment 100 includes CSPI 101, which provides services and resources that customers can use to sign up and build their own virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to its subscribing customers. Data centers within CSPI 101 may be organized as one or more regions. One exemplary region, “Region US” 102, is shown in FIG. 1 . A customer has configured a customer VCN 104 for region 102. The customer may deploy various compute instances into VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0079] In the embodiment shown in FIG. 1 , customer VCN 104 includes two subnets, “Subnet 1” and “Subnet 2,” each with its own CIDR IP address range. In FIG. 1 , Subnet 1’s overlay IP address range is 10.0 / 16, and Subnet 2’s address range is 10.1 / 16. VCN virtual router 105 represents the VCN’s logical gateway, enabling communication between subnets in VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet in VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 to Subnet 1 and a port with IP address 10.1.0.1 to Subnet 2.

[0080] Multiple compute instances may be deployed in each subnet, and the compute instances may be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines within CSPI 101. A compute instance joins a subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet 1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet 1 via a VNIC associated with C2. Similarly, multiple compute instances (which may be virtual machine instances or bare metal instances) may be part of subnet 1. Each compute instance is assigned a private overlay IP address and a MAC address via its associated VNIC. For example, in Figure 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet 1 (including compute instances C1 and C2) has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port in VCN VR105 for Subnet 1.

[0081] Subnet2 may have multiple compute instances deployed, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of Subnet2 via VNICs associated with each of the compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet2 (including compute instances D1 and D2) has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port in VCN VR105 for Subnet2.

[0082] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic between subnets of the VCN.

[0083] A particular compute instance deployed in VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet 1), endpoints on different subnets within the same VCN (e.g., communication between a compute instance in Subnet 1 and a compute instance in Subnet 2), endpoints in different VCNs in the same region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet 1 and an endpoint in the service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in a different region 108). Additionally, compute instances in a subnet hosted by CSPI 101 may communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in a customer's on-premise network 116, endpoints in other hosted remote cloud networks 118, public endpoints 114 accessible over a public network such as the Internet, and other endpoints.

[0084] Communication between compute instances on the same subnet is facilitated through the use of VNICs associated with the source and destination compute instances. For example, compute instance C1 on Subnet 1 may want to send a packet to compute instance C2 on Subnet 1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing encapsulation / decapsulation functions on the packet as needed, and forwarding / routing the packet to the next hop to facilitate communication of the packet to its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing, which then forwards the packet to the destination compute instance via the VNIC associated with the destination compute instance.

[0085] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, this communication is facilitated by the VNICs and VCN VRs associated with the source and destination compute instances. For example, in Figure 1, if compute instance C1 in Subnet 1 wants to send a packet to compute instance D1 in Subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR using the default route or port 10.0.0.1 of VCN VR105. VCN VR105 is configured to route the packet to Subnet 2 using port 10.1.0.1. Once the packet is received and processed by the VNIC associated with D1, this VNIC forwards the packet to compute instance D1.

[0086] When a packet is sent from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. VCN 104 may have one or more types of gateways associated with it. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. In this way, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. A VCN may be configured with various different types of gateways to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may be over a public network (e.g., the Internet) or a private network. Various communication protocols may be used for these communications.

[0087] For example, compute instance C1 may want to communicate with an endpoint outside VCN 104. The packet may initially be processed by a VNIC associated with source compute instance C1. The VNIC's processing determines that the packet's destination is outside of Subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR105 in VCN 104. VCN VR105 then processes the packet and, as part of that processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR105 may then forward the packet to the identified particular gateway. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by VCN VR105 to Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. Forwarding the packet from the gateway to the next hop can then facilitate the packet's delivery to its ultimate destination.

[0088] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with VCNs are also shown in FIGS. 1, 2, 3, and 4 (e.g., gateways referenced by reference numbers 134, 136, 138, 234, 236, 238, 334, 336, 338, 434, 436, and 438) and described below. As shown in the embodiment of FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with customer VCN 104 to provide a path for private network traffic communication between customer VCN 104 and another endpoint, which may be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network built using customer resources or may be a customer data center. Access to the customer on-premises network 116 is typically highly restricted. If a customer has both an on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, they may want to enable the on-premises network 116 and the cloud-based VCNs 104 to communicate with each other. This allows a customer to build an extended hybrid environment that includes the on-premises network 116 and the customer's VCN 104 hosted by CSPI 101. This communication is made possible by DRG 122. To enable such communication, a communication channel 124 is set up so that one endpoint of the channel resides in the customer on-premises network 116 and the other endpoint resides in CSPI 101 and is connected to the customer VCN 104. The communication channel 124 may traverse a public or private communication network, such as the Internet.A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. The device or equipment in the customer on-premises network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as CPE 126 shown in Figure 1. On the CSPI 101 side, the endpoint may be a host machine running DRG 122.

[0089] In one embodiment, remote peering connections (RPCs) can be added to a DRG, allowing customers to peer one VCN with another VCN in a different region. Using such RPCs, a customer VCN 104 can connect to a VCN 108 in another region using a DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft Azure cloud or the Amazon AWS cloud.

[0090] 1, an Internet Gateway (IGW) 120 may be configured for customer VCN 104, allowing compute instances on VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. IGW 120 allows a public subnet within a VCN, such as VCN 104, to directly access a public endpoint 112 on public network 114, such as the Internet, if the resources in that public subnet have public overlay IP addresses. Using IGW 120, connections can be initiated from a subnet within VCN 104 or from the Internet.

[0091] A network address translation (NAT) gateway 128 configured for a customer's VCN 104 may enable cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, but the NAT gateway 128 does not expose these resources to direct inbound Internet connections (e.g., L4-L7 connections). This allows private subnets within the VCN (such as Private Subnet 1 of VCN 104) to privately access public endpoints on the Internet. The NAT gateway only allows connections to be initiated from the private subnet to the public Internet, not from the Internet to the private subnet.

[0092] In some embodiments, a service gateway (SGW) 126 may be configured for customer VCN 104, providing a pathway for private network traffic between VCN 104 and service endpoints supported in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. One example of such a service network is Oracle's Services Network, which offers a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to a service endpoint (e.g., Object Storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections can be initiated only from subnets within the VCN, not from service network 110. When a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Additionally, resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can use a service gateway configured for that VCN.

[0093] In one embodiment, SGW 126 uses the concept of a service CIDR (Classless Inter-Domain Routing) label, which is a string that represents all of the regional public IP address ranges for a service or services of interest. A customer uses the service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. A customer can optionally use the service CIDR label when configuring security rules without having to adjust if the service's public IP address changes in the future.

[0094] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 to enable the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, but without traffic traversing a public network such as the Internet or routing traffic to the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common method used to establish network connectivity between different applications or infrastructure management functions.

[0095] Service providers (e.g., providers of services in service network 110) can provide access to services using various access models. In a public access model, services are exposed as public endpoints publicly accessible by compute instances in the customer VCN over a public network, such as the Internet, and / or they can be privately accessible through SGW 126. In a specific private access model, services are accessible as private IP endpoints in a private subnet of the customer VCN. This is called private endpoint (PE) access, and service providers can expose their services as instances in the customer's private network. Private endpoint resources represent services within the customer's VCN. Each PE appears as a VNIC in a customer-selected subnet in the customer's VCN (called a PE-VNIC and having one or more private IPs). Thus, the PE provides a way to present services in the private customer VCN subnet using VNICs. Because the endpoints are exposed as VNICs, all functionality associated with a VNIC, such as routing rules and security lists, is available to the PE VNIC.

[0096] Service providers may register their services to enable access through the PE. Providers can associate policies with services that restrict the visibility of the service to the customer tenancy. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) representing the same service.

[0097] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110) that acts as the ingress / egress point for all traffic to customer subnet private endpoints. The PAGW 130 allows providers to scale the number of PE connections without using their internal IP address resources. A provider need only configure one PAGW for any number of services registered in a single VCN. A provider can represent services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC does not appear to be attached to the customer's instance, but rather to the service with which the customer wants to interact. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0098] The use of the PE concept also allows traffic to flow across FastConnect / IPsec links and private endpoints in the customer VCN, extending private access for services to the customer's on-premises network and data center, and allows traffic to flow between LPG 132 and PEs in the customer VCN, extending private access for services to the customer's peering VCN.

[0099] Customers can control VCN routing at the subnet level, allowing them to specify which subnets in their VCNs (such as VCN 104) use each gateway. The VCN's route tables are used to determine whether traffic is allowed from a VCN through a particular gateway. For example, in a particular instance, the route table for a public subnet in customer VCN 104 may direct non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 may direct traffic destined for CSP services through SGW 126. All other traffic may be directed through NAT gateway 128. Route tables only control traffic leaving a VCN.

[0100] Traffic entering the VCN via the gateway through an inbound connection is controlled using security lists associated with the VCN. All resources in a subnet use the same route table and security lists. Security lists can be used to control the specific types of traffic that can enter and exit instances in a VCN's subnets. Security list rules can include input (inbound) and output (outbound) rules. For example, an input rule can specify an allowed source address range, while an output rule can specify an allowed destination address range. Security rules can specify a specific protocol (e.g., TCP, ICMP), a specific port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the instance's operating system can enforce its own firewall rules in line with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0101] Access from a customer VCN (i.e., by resources or compute instances deployed in VCN 104) can be categorized as public access, private access, or dedicated access. Public access describes an access model in which public IP addresses or NATs are used to access public endpoints. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In one embodiment, CSPI 101 enables customer VCN workloads with private IP addresses to access (public service endpoints of) services using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoints of the services that reside outside the customer's private network.

[0102] CSPI may also provide dedicated public access by using technologies such as FastConnect public peering, which allows customer on-premises instances to use a FastConnect connection to access one or more services in the customer VCN without traversing a public network such as the Internet. CSPI may also provide dedicated private access by using FastConnect private peering, which allows customer on-premises instances with private IP addresses to access workloads in the customer's VCN using a FastConnect connection. FastConnect is an alternative network connection using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, flexible, and economical way to create a dedicated private connection with higher bandwidth options, higher reliability, and a consistent networking experience compared to Internet-based connections.

[0103] FIG. 1 and the accompanying discussion above describe various virtualized components in an exemplary virtual network. As noted above, a virtual network is built on an underlying physical or infrastructure network. FIG. 2 is a simplified architectural diagram of the physical components in a physical network within CSPI 200 that forms the basis of the virtual network, according to one embodiment. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who have signed up for one or more services offered by the CSP. Customers are provisioned with a subset of CSPI 200's resources (e.g., compute, memory, and network resources) based on the services for which they subscribe. Customers can then build their own cloud-based (i.e., CSPI-hosted), customizable private virtual networks using the physical compute, memory, and network resources provided by CSPI 200. As previously mentioned, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances can be in the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available environment.

[0104] In the exemplary embodiment shown in FIG. 2, the physical components of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and physical networks (e.g., 218) and switches of physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of the VCN. The compute instances can include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be executed by a single host machine or by multiple different host machines. Additionally, the physical host machines can host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 1 may be executed by the NVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by a host machine and / or NVD shown in FIG.

[0105] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and makes available a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. The hypervisor on a host machine may create, run, and manage one or more compute instances on the host machine. The hypervisor on a host machine enables the sharing of the host machine's physical computing resources (e.g., compute, memory, and network resources) among the various compute instances that the host machine runs.

[0106] For example, as shown in FIG. 2, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented in software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on a host machine's operating system (OS) and runs on the host machine's hardware processor. A hypervisor provides a virtualized environment by enabling the sharing of the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) among various virtual machine compute instances that the host machine executes. For example, in FIG. 2, hypervisor 260 may reside on the OS of host machine 202 and enable the sharing of the host machine's computing resources (e.g., processing, memory, and networking resources) among the compute instances (e.g., virtual machines) that the host machine executes. A virtual machine may have its own operating system (referred to as a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. In this manner, the hypervisor allows multiple operating systems to run side by side with each other while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0107] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0108] In some cases, an entire host machine is provisioned for a single customer, and the one or more compute instances (virtual machine or bare metal instances) that it hosts all belong to that same customer. In other cases, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be part of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted on bare metal servers that do not include a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0109] As previously mentioned, each compute instance that is part of a VCN is associated with a VNIC, which may make the compute instance a member of a subnet of the VCN. A VNIC associated with a compute instance facilitates transmission of packets or frames to the compute instance. A VNIC is associated with the compute instance when the compute instance is created. In one embodiment, for compute instances executed by a host machine, the VNIC associated with the compute instance is implemented by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268, which is associated with VNIC 276, which is implemented by NVD 210 connected to host machine 202. In another example, host machine 206 hosts bare metal instance 272, which is associated with VNIC 280, which is implemented by NVD 212 connected to host machine 206. In yet another example, the VNIC 284 is associated with a compute instance 274 that the host machine 208 executes, and the VNIC 284 is executed by an NVD 212 that is connected to the host machine 208 .

[0110] For compute instances hosted by a host machine, the NVD connected to that host machine also runs VCN VRs corresponding to the VCNs of which the compute instances are elements. For example, in the embodiment shown in Figure 2, NVD 210 runs VCN VR 277 corresponding to the VCN of which compute instance 268 is an element. NVD 212 may also run one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0111] A host machine may include one or more network interface cards (NICs) that allow the host machine to connect to other devices. A NIC on a host machine may provide one or more ports (or interfaces) that allow the host machine to communicate with other devices. For example, a host machine may be connected to an NVD by one or more ports (or interfaces) provided on the host machine and the NVD. A host machine may also be connected to other devices, such as another host machine.

[0112] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0113] The NVDs, in turn, are connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also referred to as a switch fabric). In one embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, using links 228 and 230. In one embodiment, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may also be referred to as a rack.

[0114] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In one embodiment, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier, multi-node physical switching network 218. Various Clos network configurations are possible, including, but not limited to, a 2-tier network, a 3-tier network, a 4-tier network, a 5-tier network, and generally an "n" tier network. An example of a Clos network is shown in FIG. 5 and described below.

[0115] A variety of different connection configurations are possible between host machines and NVDs, including one-to-one, many-to-one, and one-to-many configurations. In one embodiment of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD 210 via NIC 232 on host machine 202. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0116] In a one-to-multiple configuration, one host machine is connected to multiple NVDs. FIG. 3 shows an example of a CSPI 300 in which a host machine is connected to multiple NVDs. As shown in FIG. 3, a host machine 302 includes a network interface card (NIC) 304 including multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices of a multi-tier physical network 318 .

[0117] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path through the TOR switch 314 to the NVD 310 and thus to the host machine 302, and a second path through the TOR switch 316 to the NVD 312 and thus to the host machine 302. These separate paths may increase the availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., if a link on one of the paths goes down) or if there is a problem with one of the devices (e.g., if a particular NVD is not functioning), the other path may be used to communicate with the host machine 302.

[0118] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may have multiple NICs allowing connection to multiple NVDs.

[0119] Referring again to Figure 2, an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0120] The NVD may be implemented in a variety of different forms. For example, in one embodiment, the NVD is implemented as an interface card with an embedded processor, referred to as a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202 and host machines 206 and 208, respectively.

[0121] However, a smart NIC is merely one example of an implementation of an NVD. Various other implementations are possible. For example, in some other embodiments, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD may be performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, enabling the TOR switch to perform various complex packet transformations used in public clouds. A TOR performing the functions of an NVD may also be referred to as a smart TOR. In still other embodiments, if virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other embodiments, some of the NVD's functions may be offloaded to a centralized service running on a group of host machines.

[0122] In some embodiments, such as when implemented as a smart NIC as shown in FIG. 2, an NVD may have multiple physical ports that allow the NVD to connect to one or more host machines and one or more TOR switches. Ports on an NVD may be classified as host-side ports (also referred to as "south ports") or network-side or TOR-side ports (also referred to as "north ports"). A host-side port of an NVD is a port used to connect the NVD to a host machine. Examples of host-side ports in FIG. 2 include port 236 on NVD 210 and ports 248 and 254 on NVD 212. A network-side port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-side ports in FIG. 2 include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 by link 228, which extends from port 256 of the NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 by a link 230 extending from a port 258 of the NVD 212 to the TOR switch 216 .

[0123] An NVD can receive packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-side port, perform necessary packet processing, and then forward the packets and frames to a TOR switch via a network-side port of the NVD. An NVD can receive packets and frames from a TOR switch via a network-side port of the NVD, perform necessary packet processing, and then forward the packets and frames to a host machine via a host-side port of the NVD.

[0124] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (LAG) consisting of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG may operate in full-duplex mode at the same speed. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG goes down, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link provides higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic may be load-balanced across the multiple physical links in the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0125] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, and functions for facilitating routing and forwarding of packets to compute instances in the VCN. In one embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining how to forward or route the packet. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with the overlay network, such as running VNICs associated with compute instances in the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing in the virtual network, running a gateway (e.g., a local peering gateway), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0126] In one embodiment, the NVD's packet processing data path may include multiple packet pipelines, each consisting of a series of packet transformation stages. In one embodiment, upon receipt of a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed linearly, one stage at a time, until it is either discarded or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header validation, throttling enforcement, new Layer 2 header insertion, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0127] The NVD may perform both control plane and data plane functions corresponding to the VCN's control and data planes. Examples of the VCN control plane are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below. Examples of the VCN data plane are shown in Figures 1, 2, 3, and 4 (see reference numbers 118, 218, 318, and 418) and described below. Control plane functions include functions used to configure the network (e.g., routing and route table configuration, VNIC configuration, etc.) and control how data is forwarded. In one embodiment, a VCN control plane is provided that centrally computes and exposes all overlay-to-foundation mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, and IGWs). Firewall rules may also be exposed by the same mechanism. In one embodiment, the NVD retrieves only mappings relevant to the NVD. The data plane functions include functionality for the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is achieved by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation functionality is achieved on the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0128] As described above, the NVD performs various virtualization functions, including VNICs and VCN VRs. The NVD may execute VNICs associated with compute instances hosted by one or more host machines connected to the VNIC. For example, as shown in FIG. 2, NVD 210 executes the functions of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206 and VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances that belong to different VCNs belonging to different customers, and the NVDs connected to the host machines can execute VNICs (i.e., perform VNIC-related functions) corresponding to these compute instances.

[0129] The NVDs also execute VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, a VCN VR corresponding to a given VCN is executed by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may execute VCN VRs corresponding to the different VCNs.

[0130] In addition to VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate the various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” shown in FIG. 2 . For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For example, the packet processing component of the NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received and communicated using the NVD and to store network information. The network information may include, for example, network flow information and per-flow information (e.g., per-flow statistics) that identify the various network flows processed by the NVD. In one embodiment, network flow information may be stored per VNIC. The packet processor may perform per-packet operations as well as implement a stateful NAT and an L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly the status and health of other components connected to the NVD.

[0131] FIG. 1 illustrates components of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed to the subnets, VNICs associated with the compute instances, VRs in the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 1 may be executed or hosted by one or more of the physical components illustrated in FIG. 2. For example, compute instances in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For compute instances hosted by a host machine, the VNICs associated with the compute instances are typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality for a VCN is performed by all NVDs connected to host machines that host or execute compute instances that are part of the VCN. The gateways associated with a VCN may be executed by one or more different types of NVDs. For example, one gateway may be executed by a smart NIC, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0132] As mentioned above, a compute instance in a customer VCN can communicate with a variety of different endpoints, which can be in the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or outside the VCN of the source compute instance. These communications are facilitated through the use of VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0133] For communication between two compute instances on the same subnet in a VCN, this communication is facilitated through the use of VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or different host machines. A packet originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the packet's destination endpoint is in the same subnet, the VNIC associated with the source compute instance forwards the packet to the NVD running the VNIC associated with the destination compute instance. The NVD then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0134] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD processes the packet using a packet processing pipeline, which may include running one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also referred to as VNIC execution). The function executed by the VNIC may include verifying the VLAN tag on the packet. Because the packet's destination is outside the subnet, the NVD then invokes a VCN VR function to execute it. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or may be running on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0135] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which forwards the packet to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be running on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may be running on a different NVD. The gateway may be running on the NVD, which may be a smart NIC, a host machine, or another implementation of the NVD. The packet is then processed by the gateway and forwarded to a next hop that facilitates delivery of the packet to the intended destination endpoint. For example, in the embodiment shown in FIG. 2, a packet originating from compute instance 268 may be sent from host machine 202 to NVD 210 over link 220 (using NIC 232). In NVD 210, VNIC 276 is called VNIC 276 because it is associated with source compute instance 268. VNIC 276 is configured to examine information encapsulated in the packet, determine a next hop to forward the packet to facilitate delivery of the packet to the intended destination endpoint, and then forward the packet to the determined next hop.

[0136] Compute instances deployed in a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. CSPI 200-hosted endpoints can include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communication between CSPI 200-hosted endpoints can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 (i.e., external to CSPI 200). Examples of such endpoints include endpoints within a customer's on-premises network or data center or public endpoints accessible over a public network such as the Internet. CSPI 200's communication with external endpoints can be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or over a private network (not shown in FIG. 2) using various communication protocols.

[0137] The architecture of CSPI 200 shown in FIG. 2 is illustrative only and is not intended to be limiting. Variations, substitutions, and improvements are possible in alternative embodiments. For example, in some implementations, CSPI 200 may include more or fewer systems or components than those shown in FIG. 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0138] FIG. 4 illustrates connections between host machines and an NVD to provide I / O virtualization to support multitenancy, according to one embodiment. As shown in FIG. 4, host machine 402 runs hypervisor 404, which provides a virtualized environment. Host machine 402 runs two virtual machine instances: VM1 406, which belongs to customer / tenant #1, and VM2 408, which belongs to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is associated with a VNIC that NVD 412 runs. In the embodiment of FIG. 4, VM1 406 is associated with VNIC-VM1 420, and VM2 408 is associated with VNIC-VM2 422.

[0139] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is configured to associate with and work together on its own logical NIC. For example, VM1 406 is associated with logical NIC A 416, and VM2 408 is associated with logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, because of the logical NICs, each tenant's virtual machine sees itself as having its own host machine and NIC.

[0140] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is sent from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Similarly, when a packet is sent from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Thus, a packet 424 sent from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. On the NVD, for a packet 424 received from a host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. According to the configuration shown in Figure 4, each tenant's compute instance can be recognized as having its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0141] FIG. 5 is a simplified block diagram of a physical network 500, according to one embodiment. The embodiment shown in FIG. 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multistage or multi-layer switching network, with possible numbers of stages or layers, such as two, three, four, or five. The embodiment shown in FIG. 5 is a three-layer network, including layers 1, 2, and 3. TOR switch 504 represents a layer 0 switch in a Clos network. One or more NVDs are connected to a TOR switch. A layer 0 switch is also referred to as an edge device of the physical network. A layer 0 switch is connected to a layer 1 switch, also referred to as a leaf switch. In the embodiment shown in FIG. 5, a set of “n” layer 0 TOR switches are connected to a set of “n” layer 1 switches, collectively forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches in that pod, but there are no switch connections between pods. In one embodiment, two pods are referred to as a block. Each block is served by or connected to a set of “n” layer 2 switches (sometimes referred to as spine switches). There may be multiple blocks in a physical network topology. The layer 2 switches are then connected to “n” layer 3 switches (sometimes referred to as super spine switches). Packet communication across the physical network 500 is typically performed using one or more layer 3 communication protocols. High availability is typically achieved by having n-way redundancy at all layers of the physical network except the TOR layer. To enable scaling of the physical network, policies may be specified for pods and blocks to control the visibility of switches to each other in the physical network.

[0142] A Clos network is characterized by a fixed maximum hop count for reaching another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch). For example, in a three-tier Clos network, a packet requires at most seven hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Similarly, in a four-tier Clos network, a packet requires at most nine hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for intra- and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to various tiers and by increasing the number of links between switches in adjacent tiers.

[0143] In one embodiment, each resource in CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used when managing the resource, for example, through a console or API. An exemplary syntax for a CID is as follows: ocid1.<RESOURCE TYPE> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: ocid1 is a string indicating the version of the CID, RESOURCE TYPE is the type of resource (for example, instance, volume, VCN, subnet, user, group, etc.), REALM is the realm in which the resource resides (example values ​​are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal cloud realm, etc.; each realm may have its own domain name); REGION is the region in which the resource resides (this part may be blank if region is not applicable to the resource), FUTURE USE is reserved for future use, UNIQUE ID is the unique part of the ID (the format can vary depending on the type of resource or service).

[0144] Virtual Private Label Cloud (vPLC) Described herein are techniques for enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs") in a region provided by a cloud service provider (CSP) to provide CSP-supplied cloud services to CSP customers. A vPLC created according to the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be created for a reseller who is a customer of the CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual datacenter and associated with a realm different from the realm associated with the cloud infrastructure in the CSP-supplied region.

[0145] A cloud service provider (CSP) provides infrastructure (referred to as cloud service provider infrastructure, or CSPI, or CSP-delivered infrastructure) used to deliver one or more cloud services to the CSP's customers. The CSP's customers may include the CSP's and / or its reseller's direct customers. The CSPI may include physical and virtual infrastructure components. Physical CSPI may include a physical infrastructure or underlying network, network virtualization devices, and interconnected high-performance compute resources, including racks, servers and host machines, memory resources, and network resources used to configure other resources. Virtual CSPI components may include components in an overlay network, such as virtual machines, virtual routers and gateways, and other virtual components.

[0146] A CSP may offer various types of cloud services. These may include various types of services such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In some embodiments, as described in this disclosure, a CSP may offer a "vPLC cloud service." A customer of a CSP may subscribe to one or more cloud services offered by the CSP. A customer may be any entity, such as an individual, an organization, or a business.

[0147] Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure. In one embodiment, a vPLC as a Service is a type of IaaS service in which infrastructure is allocated to create a virtual private label cloud for a customer. For example, if a CSP customer is a reseller, the CSP can provide a reseller-supplied cloud service using a vPLC created for the reseller.

[0148] When a customer registers or subscribes to a cloud service offered by a CSP, a tenancy (or account) is created for the customer. This account / tenancy allows the customer to access one or more registered cloud resources associated with the account / tenancy. Each tenancy is identified by a tenancy identifier (T_ID) that is assigned to the tenancy and uniquely identifies the tenancy. The tenancy identifier is generated when the tenancy is created and associated with the tenancy. For example, when a direct customer of a CSP registers for a service offered by a CSP, a tenancy is created for the customer and a tenancy identifier that uniquely identifies the tenancy is generated. When a reseller registers for a vPLC service, a tenancy is created for the reseller and a tenancy identifier that uniquely identifies the tenancy is generated. A tenancy may be a logical and secure compartment that contains all the resources and services used by the customer associated with the tenancy.

[0149] Resources in a CSP-provided infrastructure are typically identified using a unique resource identifier (resource ID or RID). Because these resources are provided in the cloud and used to provide cloud services, resource IDs are also referred to as cloud resource identifiers (or cloud IDs or CIDs). For example, in a cloud environment provided by Oracle Corporation, resources are identified using an "ocid" or "Oracle cloud identifier." In some embodiments, a resource ID is a globally unique identifier string that identifies a cloud resource. In some embodiments, a tenancy ID is a type of cloud resource ID (or cloud identifier (CID)).

[0150] A resource or cloud ID (CID) may be represented in the following format, as an example:

[0151] RID_version. <resourcetype> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: RID_version indicates the version of the resource ID.

[0152] "Resource type" identifies the particular kind of resource that the resource ID refers to. Examples of different resource types include an instance, a virtual network subnet, a tenancy, etc. For example, an "instance RID" identifies a compute instance.

[0153] "Realm" and "region" indicate the location (realm and region) of the resource. The "unique ID" is the unique part of the ID (e.g., a unique string to identify a particular resource). Thus, the tenancy ID can identify a unique ID within a CID where the resource type is tenancy.

[0154] When resources are created, a CID is generated for them. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, access control lists (ACLs), and load balancers are examples of cloud resources and can be referenced by their respective CIDs. A tenancy is a logical container used to group other cloud resources that belong to the same cloud customer account.

[0155] When an entity becomes a customer of a CSP and subscribes to the vPLC service, a vPLC is created for the customer and associated with the customer's tenancy. For example, if the customer is a reseller, a vPLC is created for the reseller and associated with the reseller's tenancy. A vPLC created for a reseller is treated as a resource and is assigned a unique resource identifier called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy. As described herein, for a vPLC created for a reseller, the vPLC ID for that vPLC is used to identify all resources and requests associated with the vPLC. Each vPLC ID uniquely identifies a vPLC.

[0156] For example, in one embodiment, all resources assigned to a vPLC are tagged with the vPLC identifier of that vPLC. In one embodiment, a vPLC ID field is inserted into the representation of the CID described above. If a resource is associated with a vPLC, the vPLC ID corresponding to that vPLC may be provided in this CID field; if a resource is not associated with a vPLC, the field may be left blank.

[0157] For the purposes of this disclosure, the following terminology will be used for the sake of clarity: CSP.Cn (or just Cn) identifies a direct (non-reseller) customer of the CSP. For example, "CSP.C1" represents the CSP's non-reseller direct customer C1, and "CSP.C2" represents the CSP's non-reseller direct customer C2.

[0158] CSP.Rn (or just Rn) identifies a reseller customer of the CSP. For example, "CSP.R1" represents CSP's reseller customer R1, and "CSP.R2" represents CSP's reseller customer R2.

[0159] In Rn.Cm, a reseller may have its own customers. This identifies a particular customer of a particular reseller. For example, "R1.C1" represents customer C1 of reseller R1, and "R2.C1" represents customer C1 of reseller R2.

[0160] T.Cn identifies the tenancy of a CSP's non-reseller customer. For example, "T.C1" represents the tenancy of non-reseller direct customer C1.

[0161] T.Rn identifies the reseller's tenancy. For example, "T.R1" represents the tenancy of reseller R1.

[0162] In T.Rn.Cm, each customer of a reseller has their own tenancy. This identifies the tenancy associated with a particular customer of a particular reseller. For example, "T.R1.C1" represents the tenancy associated with reseller R1's customer C1, and "T.R2.C1" represents the tenancy associated with reseller R2's customer C1.

[0163] T_ID.T.Cn identifies the tenancy identifier that identifies the tenancy of a CSP's non-reseller customer. For example, "T_ID.T.C1" represents the tenancy ID of non-reseller direct customer C1's tenancy.

[0164] T_ID.T.Rn identifies the tenancy identifier of the reseller's tenancy. For example, "T_ID.T.R1" represents the tenancy identifier of reseller R1's tenancy.

[0165] T_ID.Rn.Cm identifies the tenancy identifier of the tenancy associated with a particular customer of a particular reseller. For example, "T_ID.T.R1.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R1, and "T_ID.T.R2.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R2.

[0166] vPLC.Rn identifies a vPLC created for reseller Rn. For example, "vPLC.R1" identifies the tenancy created for and associated with reseller R1. This identifies a specific vPLC created for a reseller and associated with the reseller's tenancy.

[0167] vPLC_ID.vPLC.Rn identifies the vPLC identifier of a vPLC generated for a particular reseller. For example, "vPLC_ID.vPLC.R1" represents the vPLC identifier of a vPLC generated for reseller R1.

[0168] As described herein, a vPLC is created for a reseller, who is a customer of a CSP. The vPLC is associated with the reseller's tenancy. In some embodiments, a vPLC ID identifying a vPLC may be associated with a tenancy ID identifying the reseller's tenancy. This allows for the identification of a corresponding vPLC and the reseller for which the vPLC was created (including the reseller's tenancy and tenancy identifier), given a vPLC ID.

[0169] CSPI or CSP-provided infrastructure may be organized as realms, regions, and data centers. A region represents a local geographic area that includes one or more connected data centers. Regions are independent of other regions and may be separated by large distances, for example, across countries or continents. A set of application programming interfaces (APIs) is provided to the CSP-provided infrastructure in a region, and these APIs are used to perform operations using the infrastructure in the region. Operations that may be performed using the infrastructure in a region may include creating resources of the infrastructure in the region, accessing resources of the infrastructure in the region, performing operations involving resources of the infrastructure in the region, deleting resources from the infrastructure in the region, and performing other operations involving the infrastructure in the region.

[0170] A set of APIs is region-specific. Thus, a CSP-provided infrastructure in a region is characterized by a set of APIs usable with the infrastructure in that region. A CSP-provided infrastructure in a different region has a different set of APIs provided for the different region. Thus, for purposes of this application, the phrase "CSP-provided infrastructure in a region" or "CSP-provided infrastructure in a region" means that the infrastructure is characterized by a specific set of APIs provided for the region. Thus, a first set of APIs may be provided for a first region, a second set of APIs may be provided for a second region different from the first region, a third set of APIs may be provided for a third region different from the first and second regions, and so on. In this way, a specific set of APIs identifies a CSP-provided infrastructure in a particular region. In other words, a CSP-provided infrastructure in two separate regions would have two separate sets of APIs. In this manner, operations involving regional infrastructure are performed by characterizing a region with infrastructure in that region and a set of APIs associated with that region. CSP-provided infrastructure in a region is also referred to as "regional infrastructure" or "region-specific infrastructure."

[0171] For example, a CSP may provide a first infrastructure in Seattle and a second infrastructure in Portland. For the first infrastructure in Seattle, a first set of APIs may be provided, which may be characterized by the following format:

[0172] <service_identifier> .us-seattle.oci.oraclecloud.com For the second infrastructure in Portland, a second set of APIs may be provided, which may be characterized by the following format:

[0173] <service_identifier> .us-portland.oci.oraclecloud.com Note that Seattle and Portland are used here only as examples. The geographic locations of the regions can vary. For example, there can be more than one region in the same geographic city. For example, there can be a first region in a first building in a city and a second region in a second, separate building in the same city. As yet another example, there can be a first region on one floor of a building and a second region on another floor of the building. As yet another example, for infrastructure provided by a CSP on a floor, a first portion of the infrastructure can reside in a first region associated with a first set of APIs, and a second portion of the infrastructure can reside in a second region associated with a second set of APIs that is different from the first set of APIs.

[0174] CSP-provided infrastructure in a region may be organized as one or more datacenters. For example, in a particular region where a CSP may have two buildings each hosting infrastructure used to provide cloud services, the infrastructure in the first building may be organized and referred to as a first datacenter, and the infrastructure in the second building may be organized and referred to as a second datacenter. As another example, in a particular region where a CSP may have a single building with multiple floors each hosting infrastructure used to provide cloud services, the infrastructure on the first floor may be organized and referred to as a first datacenter, and the infrastructure on the second floor may be organized and referred to as a second datacenter. As yet another example, in a particular region where a CSP may have infrastructure used to provide cloud services, a first portion of the infrastructure may be organized and referred to as a first datacenter, and a second, separate portion of the infrastructure may be organized and referred to as a second datacenter.

[0175] A region may have one or more data centers. In some embodiments, all regional infrastructure is contained in a single data center, while in other embodiments, the infrastructure in a region may be organized as multiple data centers. Each data center may include infrastructure resources provided by a CSP, such as compute, storage, and networking resources. In a region, the data centers in the region may be organized as one or more availability domains (ADs). Availability domains are isolated from each other, fault-tolerant, and highly unlikely to fail simultaneously. ADs are configured so that a failure in one AD in a region is unlikely to affect the availability of other ADs in the same region.

[0176] A realm represents a logical collection of one or more regions. A realm may contain one or more regions. Realms are typically isolated from one another and do not share data. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). The realm's identity and trust profile are configured to prevent inter-realm communication. In this manner, each realm represents an isolated domain. For example, one realm may be created for government entities that are customers of a CSP, another realm may be created for commercial customers of the CSP, yet another realm may be created for private customers of the CSP, and so on. With respect to regions within a realm, regional infrastructures can communicate with each other, while regional infrastructure associated with a first realm cannot communicate with regional infrastructure associated with a second, different realm.

[0177] Resellers of cloud services can be entities such as businesses (eg, telecommunications companies), system integrators, government agencies, IT departments of large corporations, schools, and the like.

[0178] FIG. 6 is a simplified block diagram of a distributed environment 600 including a CSP-provided infrastructure in a region, illustrating an example of a virtual private label cloud (vPLC) hosted by the CSP-provided infrastructure, according to one embodiment. The distributed environment 600 illustrated in FIG. 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some implementations, the distributed environment 600 may include more or fewer systems or components than those illustrated in FIG. 6, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0179] Figure 6 illustrates a CSP-provided infrastructure 601 in a region. The infrastructure 601 may be organized as one or more data centers. The infrastructure 601 may include memory or storage resources 622, compute resources 620, networking resources (not shown), and other physical and logical resources provided by the CSP. The storage resources 622 may include one or more volatile or non-volatile memory resources. The compute resources 620 may include one or more racks, each with one or more servers hosting one or more operating systems.

[0180] A CSP can use CSP-provided infrastructure 601 to offer one or more CSP-supplied cloud services to one or more of its direct customers and to create one or more vPLCs. In the example shown in Figure 6, two vPLCs (vPLC.R1 604 and vPLC.R2 606) have been created for resellers R1 and R2, respectively. vPLC.R1 604 is used to offer one or more reseller R1-supplied and R1-branded cloud services to one or more customers 642 of R1. vPLC.R2 606 is used to offer one or more reseller R2-supplied and R2-branded cloud services to one or more customers 644 of R2.

[0181] As shown, infrastructure 601 can be communicatively coupled via communications network 652 to computing devices used by users associated with direct customer 640 and users associated with reseller customers (e.g., users associated with reseller R1's customer 642 and users associated with reseller R2's customer 644). Communications network 652 can be of various types and can include one or more communications networks. Examples of communications network 652 include, but are not limited to, the Internet, a wide area network (WAN), a local area network (LAN), an Ethernet network, a public or private network, a wired network, a wireless network, or the like, as well as combinations thereof. Additionally, various communications protocols can be used to facilitate communications, including both wired and wireless protocols, such as the IEEE 802.XX suite of protocols, TCP / IP, IPX, SAN, AppleTalk®, Bluetooth®, and other protocols. In general, communications network 652 can include any infrastructure that facilitates communications with infrastructure 601.

[0182] 6, infrastructure 601 is divided into three securely isolated portions: (1) a first portion 602 used for CSP provisioning and providing CSP-branded cloud services to one or more direct (or non-reseller) customers 640 of the CSP, (2) a second portion allocated to vPLC.R1 604 created for reseller R1, and (3) a third portion allocated to vPLC.R2 606 created for reseller R2. In one embodiment, these three portions are securely isolated from one another.

[0183] The partitioning of resources includes partitioning of storage resources 622 and compute resources 620. For example, as shown in Figure 6, storage resources 622 include a storage portion 622a included in first portion 602 used for providing CSP-supplied cloud services to the CSP's direct customers. Storage resources 622 also include a storage portion 622b allocated to vPLC.R1 604 and a storage portion 622c allocated to vPLC.R2 606. The storage or memory portion allocated to a vPLC may include one or more volatile and / or non-volatile memory resources.

[0184] 6, compute resources 620 are also partitioned. Compute resources 620 include a compute portion 620a in first portion 602 that is used to provide CSP-supplied cloud services to the CSP's direct customers. Compute resources 620 also include a portion 620b allocated to vPLC.R1 604 and a portion 620c allocated to vPLC.R2 606.

[0185] As described above, a set of APIs is provided for accessing and performing functions related to the CSP-provided infrastructure in a region. These APIs are accessible to users associated with the CSP's direct customers 640 by using a set of endpoints 610-1 provided by the CSP. For example, a user may use one or more of the endpoints 610-1 to access and / or use compute resources 620a or memory resources 622a allocated to the infrastructure used to provide CSP-provided cloud services to the CSP's customers. The endpoints in 610-1 may represent URLs with associated fully qualified domain names (FQDNs) used to access cloud services provided by the infrastructure 601. Examples of these endpoints include: createVM.compute.us-seattle.oci.oraclecloud.com (for creating VMs) deleteVM.compute.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.us-seattle.oci.oraclecloud.com (for data storage) One or more of the endpoints 610-1 may be used by users associated with the CSP's direct customers 640 and connected to the infrastructure 601 via a communications network.

[0186] In one embodiment, a set of vPLC-specific endpoints is created for each vPLC. For example, as shown in FIG. 6, a set of endpoints 612-1 is created for vPLC.R1 and a set of endpoints 614-1 is created for vPLC.R2. A user associated with a customer 642 of reseller R1 can use endpoint 612-1 to access resources associated with vPLC.R1 and perform one or more functions or operations involving the resources assigned to vPLC.R1. A user associated with a customer 644 of reseller R2 can use endpoint 614-1 to access resources associated with vPLC.R2 and perform one or more functions or operations involving the resources assigned to vPLC.R2.

[0187] In one embodiment, a set of vPLC-specific endpoints for a vPLC is generated based on the endpoints that the CSP provides to its direct customers. For example, in FIG. 6, endpoint 612-1 for vPLC.R1 is generated based on endpoint 610-1. The endpoints generated for a vPLC may include an identifier that specifically identifies the vPLC for which the endpoint was generated. For example, given the exemplary endpoints that the CSP provides to its direct customers, endpoint 612-1 for vPLC.R1 is: createVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for creating a VM) deleteVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR1.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR1" uniquely identifies the vPLC corresponding to the endpoint.

[0188] Also, an endpoint 614-1 for vPLC.R2 may be generated based on the endpoint 610-1. For example, the endpoint 614-1 for vPLC.R2 createVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for VM generation) deleteVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR2.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR2" uniquely identifies the vPLC corresponding to the endpoint.

[0189] Because the endpoints associated with a vPLC include an identifier that identifies the particular vPLC, when infrastructure 601 receives an API call, it can analyze the endpoint call to determine whether the call is for a direct customer of the CSP or for a reseller. Furthermore, because endpoints use unique identifiers for different vPLCs, the endpoint call can also identify the particular vPLC and associated reseller. Thus, based on the analysis of the endpoint call, the endpoint request can be appropriately routed and processed.

[0190] The CSP also provides a console 610-2 for use by its direct customers 640. Users associated with the CSP's direct customers 640 can use the console 610-2 to configure, access, and manage the first portion 602 of the CSP-provided regional infrastructure. In some embodiments, the console 610-2 may provide a set of web-based graphical user interfaces (GUIs) that can be used to access and manage the CSP-provided cloud infrastructure in the region. In some embodiments, the console is a web-based application.

[0191] Also, in one embodiment, a console is provided for each vPLC. For example, as shown in FIG. 6 , a console 612-2 is provided for vPLC.R1 and a console 614-2 is provided for vPLC.R2. A user associated with a customer 642 of reseller R1 can use the console 612-2 provided for vPLC.R1 to configure, access, and manage a second portion of the CSP-provided regional infrastructure assigned to vPLC.R1 604 created for reseller R1. A user associated with a customer 644 of reseller R2 can use the console 614-2 provided for vPLC.R2 to configure, access, and manage a third portion of the CSP-provided regional infrastructure assigned to vPLC.R2 606 created for reseller R2.

[0192] In some embodiments, a reseller may register with more than one vPLC, while each vPLC may still have a set of endpoints and a console. For example, if a reseller registers with two vPLCs (vPLC1 and vPLC2), vPLC1 may have a set of endpoints (e.g., 612-1) and a console (e.g., 612-2), and vPLC2 may have another set of endpoints (e.g., 614-1) and a console (e.g., 614-2). Both vPLC1 (e.g., 604) and vPLC2 (606) may be associated with the same reseller.

[0193] The infrastructure 601 may include networking resources that enable communication with the infrastructure 601. These network resources may include, for example, a network gateway 628 that enables communication of traffic from the CSP-provided regional infrastructure 601 to other destinations, which may be communicatively coupled to the infrastructure 601 via a public (e.g., the Internet) or private network, and may reside in an on-premise network, another cloud network, etc. The gateway 628 may be implemented as a physical network device (e.g., a router), as a logical networking entity (e.g., a virtual router), or as some combination of physical and logical entities.

[0194] The infrastructure 601 may include a control plane (CP) 624 and a management plane (MP) 626 provided by the CSP. In some embodiments, the CP 624 may be responsible for providing management, deployment, and orchestration functions for the cloud environment provided by the CSP. For example, the CP may receive requests from the CSP's direct customers, reseller R1's customers, and reseller R2's customers through endpoints and consoles, collaborate with other components in the infrastructure, such as resource managers and management planes, determine whether the requests can be fulfilled, and respond accordingly to the CSP's direct customers, reseller R1's customers, and reseller R2's customers. In other words, the CSP's CP 624 may be responsible for performing vPLC-related functions. In some embodiments, the CSP may also provide a vPLC-specific control plane configured specifically to perform vPLC operations. In some embodiments, there may be one vPLC control plane that can perform processing for multiple vPLCs. In other embodiments, each vPLC may have its own CP.

[0195] In some embodiments, the MP 626 may be a collection of software components responsible for provisioning all the processes and workflows required for a requested operation. For example, for a service request to access a cloud service, the MP may provision a workflow that includes checking the identity of the requester and checking DNS records to obtain an IP address to access the service backend. The MP ensures that the various tasks are performed correctly and in the correct order by the data plane.

[0196] In one embodiment, compute resources 620 may include one or more racks each including one or more servers hosting one or more operating systems. The division of computer resources 620 into resources 620a allocated to infrastructure for serving the CSP's direct customers 640 and resources 620b and 620c allocated to vPLCs may be performed along various levels or boundaries. The division of compute resources at a particular level may be referred to herein as a segmentation level. In one embodiment, the division may be performed at the rack level (i.e., rack-level segmentation), with one or more individual racks assigned to 620, 620b, and 620c. In this embodiment, a rack is exclusively assigned to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640.

[0197] In another embodiment, the partitioning may be performed at the server level (i.e., segmented at the server level), with one or more individual servers in the same or different racks assigned to 620, 620b, and 620c. In this embodiment, a server is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a rack may have multiple servers, one server on the rack may be assigned to 620a, another server to 620b, and a third server to 620c. Thus, a rack may be shared between two vPLCs, or may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0198] In some embodiments, the partitioning is performed at the hypervisor level (i.e., segmented at the hypervisor level), and one or more individual hypervisors on the same server or different servers on the same rack or different racks may be assigned to 620, 620b, and 620c. In this embodiment, a hypervisor is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a server may host multiple hypervisors, one hypervisor on a server may be assigned to 620a, another hypervisor on the same server may be assigned to 620b, and a third hypervisor on the same server may be assigned to 620c. Thus, a server may be shared between two vPLCs or may be shared between a vPLC and the infrastructure used by a CSP to provide CSP-delivered services to its direct customers 640.

[0199] A hypervisor on a server on a rack can manage and run multiple virtual machines (VMs). In some embodiments, the partitioning is performed at the VM level (i.e., segmentation at the VM level), and VMs running with the same hypervisor can be assigned to 620, 620b, and 620c. In this embodiment, a VM is assigned exclusively to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640. Because a hypervisor can support multiple VMs, one VM may be assigned to 620a, another to 620b, and a third VM on the same hypervisor may be assigned to 620c. Thus, a hypervisor may be shared between two vPLCs, or it may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0200] In still other embodiments, a combination of the above partitioning techniques may be used. Regardless of the partitioning technique used, the partitioning is performed securely and safely so that traffic and storage destined for a particular vPLC for a particular reseller is not visible to or accessible by other resellers, their respective customers, or other direct customers of the CSP. Additionally, traffic and storage destined for a particular customer of a reseller is not visible to or accessible by other customers of that reseller.

[0201] Figure 7 illustrates the relationships between a CSP, its direct customers, its reseller customers, and their associated users, and each of the reseller's customers and their associated users, according to one embodiment. As shown in Figure 7, CSP 710 is at the top or root of the hierarchy. CSP 710 may provide one or more cloud services to which the CSP's customers may subscribe. In Figure 7, the CSP's customers are represented by 702. A tenancy, identified by a tenancy ID, is created for each registered customer of CSP 710.

[0202] Customers 702 of a CSP 710 can include direct or non-reseller customers and reseller customers. A reseller customer is an entity that subscribes to a vPLC service offered by a CSP, and a vPLC generated by the reseller as a result of the subscription to the vPLC service is used to provide reseller-supplied and reseller-branded cloud services to the reseller's customer. A reseller's customer may be different from the CSP's customers. A direct or non-reseller customer of a CSP is an entity that subscribes to one or more CSP-supplied cloud services, but does not use the CSP's infrastructure to provide any cloud services of its own to its customers.

[0203] In the example shown in Figure 7, the direct or non-reseller customers of the CSP 710 include customer C1 726 with an associated tenancy T.CSP.C1 and customer C2 728 with an associated tenancy T.CSP.C2. There may be multiple such direct customers of the CSP 710. One or more users may be each direct customer of the CSP, and these users may use one or more of the CSP-supplied cloud services to which the corresponding direct customer subscribes. For example, users associated with a customer of the CSP may be employees or agents of the customer.

[0204] 7, reseller customers of a CSP 710 include reseller R1 720 with associated tenancy T.CSP.R1 and reseller R2 722 with associated tenancy T.CSP.R2. There may be multiple such reseller customers of a CSP 710. One or more users (not shown in FIG. 7) may be each reseller customer of the CSP.

[0205] Here, a reseller may have its own customers who subscribe to one or more reseller supply and reseller brand cloud services, which are provided to the reseller using a vPLC generated by the CSP. In Figure 7, customers of reseller R1 720 are indicated using reference number 704, and customers of reseller R2 722 are indicated using reference number 706.

[0206] A reseller's customers may each have their own tenancy and associated set of users. For example, as shown in FIG. 7 , reseller R1 may have multiple customers, including customer R1.C1 740 with tenancy T.R1.C1 and associated users, customer R1.C2 742 with tenancy T.R1.C2 and associated users, and others. Reseller R1 may supply a set of R1-supplied and branded cloud services, and R1's customers can register for one or more of the R1-supplied services. The set of one or more R1-supplied services that R1.C1 740 subscribes to may be the same as or different from the set of one or more R1-supplied services that R1.C2 742 subscribes to. The cloud services supplied by R1 722 may be different from the cloud services supplied by reseller R2 722 and the cloud services supplied by CSP 710.

[0207] Reseller R2 722 may have multiple customers subscribing to R2-supplied and branded cloud services provided using the vPLC created for R2. As shown in FIG. 7 , R2's 722 customers include customer R2.C1 744 with tenancy T.R2.C1 and associated users, customer R2.C2 746 with tenancy T.R2.C2 and associated users, and others. The cloud services supplied by R2 722 may be different from the cloud services supplied by reseller R1 720 and the cloud services supplied by CSP 710. R2's customers may subscribe to one or more of the R2-supplied cloud services. The set of one or more R2-supplied services that R2.C1 744 subscribes to may be the same as or different from the set of one or more R2-supplied services that R2.C2 746 subscribes to.

[0208] As shown in Figure 7, there are two levels of tenancies that come into play when a reseller uses vPLC to sell reseller-supplied cloud services to their customers. The first level of tenancies corresponds to tenancies associated with the CSP's 710 customers (including direct customers, i.e., non-reseller customers, and reseller customers). These tenancies may also be referred to as "CSP's Customer" tenancies (or CoCT). In Figure 7, the CoCT includes tenancies associated with reseller R1 720, reseller R2 722, direct customer C1 726, and direct customer C2 728. The second level of tenancies includes tenancies associated with the reseller's customers. These tenancies may also be referred to as reseller's customer tenancies (or CoRT). In Figure 7, CoRT includes tenancies associated with R1.C1 740, R1.C2 742, R2.C1 744, R2.C2 746, etc.

[0209] 8 illustrates the relationships between resources allocated to the tenancies of the CSP's direct customers, the CSP's reseller tenancies, and the tenancies of each reseller's customers, according to one embodiment. Referring to FIG. 8, in one embodiment, a CSP-provided infrastructure (e.g., data center) 810 in a region may be divided into multiple portions that are securely isolated for the CSP's resellers, such as resources allocated to a vPLC.R1 820 created for reseller R1 and associated with reseller R1's tenancy (T.R1), and resources allocated to a vPLC.R2 822 created for reseller R2 and associated with reseller R2's tenancy (T.R2). The partitions of resources may also include partitions that are securely isolated for the CSP's non-reseller direct customers C1 and C2, such as resources 826 allocated to non-reseller customer CSP.C1's tenancy (T.CSP.C1) and resources 828 allocated to non-reseller customer CSP.C2's tenancy (T.CSP.C2). The partitions of CSP-provided regional infrastructure resources allocated to reseller R1's tenancy 820, reseller R2's tenancy 822, customer CSP.C1's tenancy, and customer CSP.C2's tenancy may be referred to as first-level resource partitions.

[0210] Each securely isolated portion of resources associated with a first-level resource partition or reseller's tenancy (i.e., the R1 tenancy and the R2 tenancy) may be assigned or tagged with a vPLC ID (i.e., resource ID), such as vPLC_ID.vPLC.R1 820 and vPLC_ID.vPLC.R2 822. Each of these CSP's non-reseller direct customer partitions may also be tagged with a resource ID. Each of the CSP's non-reseller direct customer tenancies is accessible to respective users, who can use one or more of the CSP-supplied cloud services to which each corresponding direct customer subscribes.

[0211] Within each first-level resource partition (or portion) of a reseller may be additional second-level resource partitions that are assigned to the tenancies of the reseller's customers. For example, the tenancy (T.R1.C1) associated with reseller R1's customer C1 may be assigned securely isolated resource partition (vPLC.R1.C1) 840, and the tenancy (T.R1.C2) associated with reseller R1's customer C2 may be assigned securely isolated resource partition (vPLC.R1.C2) 842. Isolation partition vPLC.R1.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1, and isolation partition vPLC.R1.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1.

[0212] Similarly, a tenancy (T.R2.C1) associated with reseller R2's customer C1 may be assigned securely isolated resource partition (vPLC.R2.C1) 844, and a tenancy (T.R2.C2) associated with reseller R2's customer C2 may be assigned securely isolated resource partition (vPLC.R2.C2) 846. The isolated partition vPLC.R2.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1, and the isolated partition vPLC.R2.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1. Each tenancy associated with a reseller's customer (e.g., R1.C1 840, R1.C2 842, R2.C1 844, or R2.C2 846) is accessible to one or more users associated with the customer who can use one or more of the reseller-supplied cloud services to which the corresponding customer subscribes.

[0213] 9A and 9B illustrate a mechanism for storing information in the tenancies of a CSP's individual reseller customers, according to one embodiment. In some embodiments, the vPLC IDs and tenancy IDs (sometimes referred to as customer tenancy IDs or CT IDs) of tenancies associated with a reseller's customers may be associated in two ways: 1) a distributed scheme (or record space per vPLC) as shown in FIG. 9A; and 7) a centralized scheme (or vPLC ID as the partition key) as shown in FIG. 9B. While FIGS. 9A and 9B show the information stored in a table format, in some embodiments the information may be stored in different formats, such as an array, a key-value store, XML, JSON, etc.

[0214] FIG. 9A illustrates a first approach, per-vPLC record space. In some embodiments, the per-vPLC record space may be an approach in which tenancy-related information for each reseller is stored in a vPLC-specific table under a vPLC ID. In other words, the information may be stored in a distributed manner. Each table is assigned a vPLC ID. In each table, one row may be designated for a customer of the reseller, another row may be designated for another customer of the reseller, and so on. For example, in FIG. 9A , there are two tables for storing information such as identification information (e.g., user logins, associated authentication information, and certificates) or billing information (e.g., resource usage, pricing). Table 910 is assigned the vPLC ID (vPLC_ID.vPLC.R1) of vPLC.R1 associated with reseller R1, and table 920 is assigned the vPLC ID (vPLC_ID.vPLC.R2) of vPLC.R2 associated with reseller R2. In table 910, there are three records for three customers C1, C2, and C3 of reseller R1. One record / row may include the tenancy ID (T_ID.T.R1.C1) of the tenancy associated with reseller R1's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R1.C2) of the tenancy associated with reseller R1's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R1.C3) of the tenancy associated with reseller R1's customer C3 and related information for customer C3.

[0215] The same structure can be used for table 920, storing three records for reseller R2's three customers C1, C2, and C3. One record / row may include the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3 and related information for customer C3.

[0216] FIG. 9B illustrates a second approach using the vPLC ID as a partition key. In FIG. 9B, the second approach stores information in a table with two columns: a customer tenancy ID and a vPLC ID. Searches are performed using the vPLC ID as a partition key. In other words, information may be centrally stored in a table. For example, in FIG. 9B, a reseller R1 (vPLC_ID.vPLC.R1) associated with a vPLC ID has three customers C1, C2, and C3, and is associated with tenancy IDs T_ID.T.R1.C1, T_ID.T.R1.C2, and T_ID.T.R1.C3 that identify the tenancies of the three customers. In the vPLC ID column, the first three rows of table 930 may be marked as vPLC_ID.vPLC.R1, which is the vPLC ID of vPLC.R1 generated for reseller R1. The customer tenancy ID column in row 1 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C1 (T_ID.T.R1.C1), while the tenancy ID column in row 2 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C2 (T_ID.T.R1.C2), and the tenancy ID column in row 3 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C3 (T_ID.T.R1.C3). The information stored in the first three rows belongs to reseller R1's customers C1, C2, and C3.

[0217] The same structure can be used to store records for reseller R2's customers C1, C2, and C3 in the same table 930, e.g., the fourth, fifth, and sixth rows. For example, in the vPLC ID column, the last three rows of table 930 may be marked as vPLC_ID.vPLC.R2, which is the vPLC ID of vPLC.R2 generated for reseller R2. The customer tenancy ID column in row 4 may be marked as the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1. Meanwhile, the tenancy ID column in row 5 may be marked as the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2, and the tenancy ID column in row 6 may be marked as the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3. The information stored in the last three rows belongs to reseller R2's customers C1, C2, and C3.

[0218] FIG. 10 is a flowchart illustrating an example of a vPLC configuration process, according to an embodiment. The process illustrated in FIG. 10 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 10 and described below is exemplary and not intended to be limiting. While FIG. 10 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 10 may include more or fewer steps than those depicted in FIG. 10.

[0219] For example, in one embodiment, the process illustrated in Figure 10 may be performed by a CSP, and some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP)), or by one or more CSP-provided cloud services, such as an endpoint management service (EMS), an identity management service, etc.

[0220] As previously described, a vPLC may be created when a reseller registers for a CSP-provided vPLC service. In step 1002, the vPLC may be created by receiving a signal. In one embodiment, the signal may be a request received when a reseller registers for a CSP-provided vPLC service. The request may be an agreement between the CSP and the reseller, listing the virtual model (e.g., nested or segmented model), database, resource placement, customization details, etc. In another embodiment, a signal may be received from some component indicating that a vPLC is being created.

[0221] Step 1004 may include receiving configuration information for the vPLC to be created. In some embodiments, the configuration information may include information about the tenancy in which the vPLC is created (e.g., a tenancy identifier) ​​and information for identifying the realm associated with the vPLC. As described above, a tenancy is created for a reseller, and a tenancy identifier that uniquely identifies the tenancy is generated for the reseller associated with the vPLC.

[0222] As mentioned above, in use cases where a vPLC is used as a virtual data center and may be associated with a different realm than the realm associated with the cloud infrastructure in the region provided by the CSP, the identity and trust profile information (e.g., passwords, credentials, etc.) of the different realm needs to be associated with the vPLC. The association of the identity and trust profile information of the different realm may be achieved by creating a mapping (or "shadow copy") of the identity and trust profile information in the vPLC. Further details are discussed in FIG. 18.

[0223] Step 1006, which includes steps 1008-1018, may involve performing a process to create a vPLC for a reseller R. Step 1008 may involve generating a vPLC identifier for the vPLC. The vPLC created for the reseller is treated as a resource and assigned a unique resource identifier, called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy.

[0224] In step 1010, a set of resources of a CSP-provided infrastructure in a region may be allocated to the vPLC. The set of resources allocated to the vPLC may be selected based on an agreement between the reseller and the CSP. For example, reseller R may subscribe to one or more cloud services offered by the CSP, such as compute services and object storage services. Thus, the set of resources may include compute resources and object storage resources.

[0225] In step 1012, a namespace may be reserved for the vPLC. The CSP's resource allocation service (or resource manager (RM)) may create and reserve a namespace that groups various types of resources that may support the vPLC. A namespace may represent a logical grouping or virtual boundary that helps organize and manage resources and prevent naming conflicts when multiple instances of similar resources are created. Thus, a namespace provides a structured way to organize and group related resources. In some embodiments, namespace 0 is reserved for the CSP. Namespaces 1 and beyond are used for the vPLC. In some embodiments, a wide range of Internet Protocol (IP) addresses may be reserved for the vPLC and associated with the namespace for traffic separation purposes.

[0226] In step 1014, a console may be generated for the vPLC that allows reseller R, the reseller's customers, and users associated with the reseller's customers to configure, access, and manage resources deployed in the vPLC, as well as visualize, access, and interact with reseller-supplied cloud services. Reseller R may also configure the console to provide different, customized experiences for different customers of the reseller.

[0227] In step 1016, a set of endpoints may be generated for the vPLC. In some embodiments, some of the set of endpoints may be default endpoints to be used (e.g., identity endpoints), and some endpoints may be determined based on the cloud services that the reseller registers. For example, the reseller may register for one or more CSP-supplied cloud services, such as compute, storage, virtual cloud network (VCN), database, etc. As a result, a default identity endpoint for accessing the identity services may be generated, as well as endpoints for accessing the registered services.

[0228] In step 1018, vPLC-related information such as the vPLC ID, the reseller's identification information, and other information facilitating the reseller-supplied cloud service may be stored in the vPLC namespace. In step 1020, the CSP may send a response indicating the created vPLC and other information to the reseller R. The reseller may then begin managing the vPLC for the reseller's customer, such as creating a customer account and configuring the reseller-supplied cloud service.

[0229] FIG. 11 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. FIG. 11 may be similar to FIG. 6, and both figures include securely isolated portions (or partitions) of the CSP-provided regional infrastructure for the CSP's direct customers, vPLC.R1 associated with reseller R1, and vPLC.R2 associated with reseller R2 (e.g., first portion 602, second portion 604, and third portion 606 of FIG. 6). The CSP's direct customers, customers of reseller R1, and customers of reseller R2 may access their respective partitions through one or more endpoints and consoles. Also, as shown in FIG. 6, the CSP-provided regional infrastructure 602 may have a control plane, a management plane, and a gateway.

[0230] However, Figure 11 shows in more detail the interconnections between resources in the CSP-provided regional infrastructure, such as compute, storage, gateways, CPs, and the management plane. Additionally, the CSP's non-reseller direct customers, customers of reseller R1, and customers of reseller R2 may access the CSP-provided regional infrastructure from the Internet or a corporate network. For example, a gateway may be used to access the Internet or a corporate network from a Real Application Cluster (RAC) in the CSP-provided regional infrastructure.

[0231] FIG. 11 illustrates three cloud partitions running on the same CSP-provided regional infrastructure 1102: the CSP's native cloud 1120 and two vPLCs (vPLC.R1 1122 and vPLC.R2 1124) for resellers R1 and R2. The CSP-provided regional infrastructure 1102 is operated by the CSP. The CSP-provided regional infrastructure 1102 may include a control plane 1104 and a management plane 1105. As previously mentioned, the control plane 1104 may be responsible for providing management, deployment, and orchestration functions for the CSP-provided cloud environment. The management plane 1105 may be a collection of software components responsible for provisioning all processes and workflows required for requested operation. The software implementing the cloud service may originate from a deployment system 1106 and be integrated into data centers in the region through the management plane 1105. Software updates (e.g., new services) to the control plane 1104 may be delivered by a deployment system 1106 through the management plane 1105. The management plane 1105 also allows the CSP administrator or reseller operator 1107 deploying the software / services to access the CSP-provided regional infrastructure.

[0232] Compute resources, namely three Real Application Clusters (RACs) 1140, 1142, and 1144, sharing the same storage 1150, are partitioned and allocated to the CSP's native cloud 1120, vPLC.R1 1122, and vPLC.R2 1124, respectively. For example, RAC 1140 is dedicated to the CSP's direct customers, RAC 1142 is designated for vPLC.R1, and RAC 1144 is for vPLC.R2.

[0233] Network resources can also be divided or shared between the CSP's direct customers and resellers to connect their respective computing resources to the Internet or corporate network. For example, each RAC 1140, 1142, and 1144 uses its network gateway 1160, 1162, and 1164 to access the Internet 1108 or corporate network 1109. In some embodiments, two RACs may share the same network interface. For example, RAC 1142 and RAC 1144 share the same gateway 1162 to access the Internet 1108.

[0234] The storage in Figure 11 is simplified. Storage 1150 may include various types of storage, such as general storage, dedicated storage, block storage, and object storage. In Figure 11, vPLC.R1 and vPLC.R2 have dedicated storage 1152 and 1154, respectively. General storage 1150 belongs to the CSP.

[0235] In FIG. 11 , a CSP's native cloud 1120 may provide CSP-supplied cloud services to the CSP's non-reseller direct customers (e.g., CSP.C1-Cn) (e.g., businesses). Users associated with the reseller's customers can access the services through dedicated service endpoints that are part of a respective namespace, including a fully qualified domain name (FQDN), IP address, console experience, identity information, etc. The CSP may provide individual consoles for each reseller, and the consoles provide the services through the endpoints. The endpoints associated with the consoles may vary depending on the services provided to the reseller's customers based on the reseller's subscription. For example, individual reseller customers may submit access requests through dedicated endpoints or consoles.

[0236] Users can use service endpoints 1170 and 1172 (two endpoints are shown, but more are possible) published by the CSP, which have DNS names. The native cloud 1120 may also have a console 1174 with a look and feel. The CSP's direct customers on the Internet 1108 can request launching or termination of instances by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1170 and 1172. Reseller R1's customers on the Internet 1108 can request launching or termination of instances by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1180 and 1182. Reseller R2's customers on the corporate network 1109 can request launching or termination of instances by accessing the CSP-provided regional infrastructure 1102 through dedicated endpoints 1190 and 1192. The control plane 1104 can then launch instances, terminate instances, set up connections for these instances, and orchestrate services in the data center. Network interfaces 1160 and 1162 (e.g., Internet Gateways (IGWs)) of the CSP and vPLC.R1, respectively, can provide network interconnection to the Internet 1108. Network interface 1164 (e.g., Dynamic Routing Gateway (DRG)) of vPLC.R2 can provide network interconnection to the enterprise network 1109.

[0237] In FIG. 11 , vPLC.R1 1122 and vPLC.R2 1124 share the same physical infrastructure. Each endpoint for each reseller (e.g., endpoints 1180 and 1182 for reseller R1 and endpoints 1190 and 1192 for reseller R2) is unique. Consoles 1184 and 1194 are customized for reseller R1 and reseller R2, respectively. The vPLC endpoints and consoles can connect to the Internet or their respective corporate networks as needed. For example, in FIG. 11 , vPLC.R1 connects to the Internet 1108 through a network gateway 1162 (e.g., IGW). However, vPLC.R2 does not connect to the Internet; instead, it connects to the corporate network 1109 through a secure network 1164 (e.g., FastConnect). 11, gateway 1162 may be sliced ​​and shared by vPLC.R1 and vPLC.R2 and may be part of an orchestration implemented by a CSP. In this example, some of vPLC.R2's data may go to the Internet 1108 and some may go to its corporate network 1109.

[0238] If the request arrives through console 1194, it may go to control plane 604. The control plane may determine that the request arrived from vPLC.R2 1124, find the available resource (e.g., computer) from the RAC specified for vPLC.R2 (i.e., 1144 in this case), and provision and activate this resource. Control plane 604 may then provide an IP address for the network configuration (e.g., VCN) of reseller R2's customer through console 1194. Control plane 1104 can configure gateway 1164 to either the enterprise network 1109 or the Internet 1108, depending on the configuration. The control plane can recognize not only the request but also the requestor (either the CSP's direct customer, a customer of reseller R1, or a customer of reseller R2).

[0239] As mentioned above, partitioning of the CSP-provided regional infrastructure can be performed for different vPLCs at various predetermined infrastructure levels (e.g., rack level, server level, hypervisor level, or VM level). To further explain, in the case of partitioning at the rack level, a vPLC may have its own dedicated rack and its internal resources. In the case of partitioning at the server level, multiple vPLCs share a rack, while each vPLC has its own dedicated server. In yet another example, in the case of partitioning at the hypervisor level, multiple vPLCs share a server, while each vPLC has its own dedicated hypervisor. In the case of partitioning at the VM level, multiple vPLCs share a hypervisor, while each vPLC has its own dedicated VM. Thus, there can be two main vPLC infrastructure partitioning models (or two ends of a spectrum) for realizing a multi-vPLC scheme: a segmented model and a nested model. The segmented model can represent resource partitioning at the rack level. The nested model can represent resource partitioning at the VM level. However, other variations between the two are possible given operational efficiency (eg, control plane sharing) and fragmentation (ie, resource division) considerations.

[0240] FIG. 12 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to one embodiment. In this model, which represents one end of the partitioning spectrum, each vPLC may have its dedicated resources (including compute and network resources) at the rack level. In other words, a rack may be exclusively assigned to a particular vPLC and not used by other vPLCs or by a CSP to provide CSP-supplied services directly to customers. In the segmentation model, resource and usage tracking is performed at the physical resource level based on affinity tags (e.g., vPLC ID and customer tenancy ID). While there may be default behavior per segment, each segment can be customized by the resellers associated with the vPLCs in that segment.

[0241] In FIG. 12 , multiple vPLCs (e.g., vPLC.R1 1210, vPLC.R2 1220, and vPLC.Rn 1230) may exist in a CSP-provided infrastructure (e.g., a data center) in a certain region. In one embodiment, in a segmentation model, there may also be physical and logical partitions, as shown in FIG. 12 . For example, in the case of physical partitions, vPLCs may be assigned dedicated hardware resources (e.g., racks), each of which may include one or more servers hosting one or more operating systems. In FIG. 12 , a first rack is assigned to the vPLC.R1 segment 1210, a second rack is assigned to the vPLC.R2 segment 1220, and a third rack is assigned to the vPLC.Rn segment 1230. Each vPLC may also have a dedicated networking switch (also referred to as a gateway) that connects to the Internet / corporate network 1260. For example, vPLC.R1 1210 has gateway 1219, vPLC.R2 1220 has gateway 1229, and vPLC.Rn 1230 has gateway 1239. Thus, in FIG. 12 , vPLC.R1 1210 has dedicated servers 1214a and 1214b, a hypervisor 1216 running on server 1214b and hosting VMs 1218a-1218n, and a gateway (e.g., IGW) 1219. Similarly, vPLC.R2 1220 has dedicated servers 1224a and 1224b, a hypervisor 1226 running on server 1224b and hosting VMs 1228a-1228n, and a gateway 1229. The vPLC.Rn 1230 includes dedicated servers 1234a and 1234b, a hypervisor 1236 that runs on the server 1234b and hosts VMs 1238a to 1238n, and a gateway 1239.

[0242] In the case of logical partitioning, there is a common control plane 1250 and resource manager 1252 for all vPLCs that are managed and operated by the CSP. However, in a segmented model, a data plane may be dedicated to each vPLC. For example, each vPLC.R1, vPLC.R2, and vPLC.Rn has its own data plane 1212, 1222, and 1232, respectively. In some embodiments, the resource manager 1252 may be part of the management plane. In other embodiments, the resource manager 1252 may be separate from the management plane.

[0243] In some embodiments, a user associated with a customer 1280 of reseller R1 may launch a VM instance or a bare metal server by sending a request through one of vPLC.R1 endpoints 1270. Assume the request is to launch a new VM instance. In this case, control plane 1250 may work with an identity service to check the user's proper credentials for vPLC.R1, and resource manager 1252 may then identify all resources dedicated to vPLC.R1, such as an IP address range. This results in a VM instance (e.g., 1218a) being launched on hypervisor 1216 in vPLC.R1 segment 1210. Control plane 1250 then sends a response back to the requesting user through vPLC.R1 endpoint 1270. If the request is for a bare metal server (e.g., 1214a), resources for vPLC.R1 may be provisioned and allocated to vPLC.R1 with all the required settings by resource manager 1252. Similar processes may be performed such that requests from users associated with customers of reseller R2 1282 are fulfilled by vPLC.R2 segment 1220, and requests from users associated with customers of reseller Rn 1284 are fulfilled by vPLC.Rn segment 1230. Each process for a vPLC may be performed independently of and concurrently with other vPLCs.

[0244] The vPLC segmentation model offers several potential advantages, including physical separation and sharing of a common control plane 1250 operated by the CSP. Additionally, a resource manager 1252 manages resources across the vPLCs 1210-1230. However, scaling challenges may arise because resources, including networking and storage, are replicated as specified for each vPLC. This model also makes small footprints, such as a single rack or server, less feasible. However, the vPLC segmentation model still allows for sharing of power, cooling, and so on. As mentioned previously, the segmentation model is only one end of the spectrum, and there are various resource- and efficiency-based partitioning methods.

[0245] FIG. 13 is a block diagram illustrating a vPLC nesting model for resource partitioning, according to one embodiment. The nesting model of FIG. 13 illustrates the other end of the resource partitioning spectrum at the VM level. In other words, a VM may be exclusively assigned to a particular vPLC and not used by other vPLCs or by the CSP to provide CSP-delivered services directly to its customers. However, hypervisors, servers, or racks may be shared between the CSP's customers and the customers of one or more vPLCs. In some embodiments, the nesting model may have a base layer (such as a hypervisor) that is invisible to the customer as a shared foundation. Abstraction layers (e.g., VMs) may also be built on top of the base layer, allowing each reseller to clone and customize the abstraction layers for their customers.

[0246] In the embodiment of FIG. 13 , all vPLCs (e.g., vPLC.R1, vPLC.R2, and vPLC.Rn) are hosted on a single segment with a shared infrastructure represented as a “server CSP” 1320 (e.g., a bare metal server) operated by a CSP, which may have control plane functionality. The server CSP 1320 may be configured to host a hypervisor 1330 that manages virtual machines (VMs) 1332a-1332n running on the hypervisor. In some embodiments, a segment may include multiple racks. Here, two VM instances for vPLC.R1 1332a and 1332b, one VM instance for vPLC.R2 1332c, and one VM instance for vPLC.Rn 1332n reside on the shared hypervisor 1330. In segment 1310, all vPLCs share the same data plane 1312. In some embodiments, a segment may include multiple racks. Since hypervisors, servers, and racks can be shared by multiple vPLCs, the servers of these vPLCs may come from different racks.

[0247] 13 further illustrates a variation of the splitting scenario in which a customer of reseller R2 may request an additional bare metal server, server vPLC.R2 1322, that is provisioned as a dedicated server. In the vPLC nesting model, a shared server pool can be placed wherever space is available, thus providing a low-cost proposition to the reseller.

[0248] In FIG. 13 , a gateway 1340 (e.g., IGW or DRG) shared by the vPLCs has a connection to the Internet or a corporate network as needed. Incoming packets from the Internet / corporate network 1360 may be tagged (i.e., encapsulated in each packet header) with vPLC-related information (e.g., a vPLC ID and a customer tenancy ID) at the gateway 1340 for tracking purposes to reach the appropriate vPLC. Packet encapsulation helps separate traffic on a single segment 1310 shared by multiple vPLCs. As an example, a packet destined for VM 1332a, which belongs to a tenancy associated with customer C1 of reseller R1, which is associated with vPLC.R1, may have vPLC_ID.vPLC.R1 and customer tenancy ID (T_ID.R1.C1) encapsulated in the packet header. The packet may then be routed to VMs 1332a and 1332b separated for vPLC.R1 based on the vPLC ID (vPLC_ID.vPLC.R1). The customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 in the packet header may further aid in routing the packet to tenant C1 running on VM 1332a, where the packet is decapsulated before delivery to VM 1332a.

[0249] Similarly, a packet destined for VM1332c, which belongs to a tenancy associated with customer C1 of reseller R2, which is associated with vPLC.R2, may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) encapsulated in the packet header. The packet can then be routed to resources carved out for vPLC.R2 based on the vPLC ID (vPLC_ID.vPLC.R2). The customer tenancy ID (T_ID.R2.C1) associated with customer C1 of reseller R2 in the packet header can further aid in routing the packet to tenant C1 running on VM1332c in vPLC.R2. The packet is decapsulated before delivery to VM1332c.

[0250] Outgoing packets may perform the reverse process. For example, an outgoing packet originating from VM 1332a in vPLC.R1 and destined for the Internet 1360 may have vPLC_ID.vPLC.R1 and a customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. Similarly, an outgoing packet originating from VM 1332c in vPLC.R2 and destined for the Internet 1360 may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) associated with reseller R2's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. As a result, both inbound and outbound traffic are separated in a single segment 1310 shared by multiple vPLCs.

[0251] In some embodiments, in a nested model, one or more VMs may belong to a direct customer of the CSP and share the same hypervisor as other vPLCs.

[0252] Reseller users accessing the vPLC infrastructure are authenticated in both the segmented and nested models. A user associated with a particular reseller's customer can access an endpoint and authenticate as a user of the vPLC associated with that particular reseller using the user's credentials (e.g., username and customer tenancy ID). For example, in some embodiments, in FIG. 13 , a user associated with reseller R1's customer C1 may launch a VM instance by sending a request through one of the vPLC.R1 endpoints 1302. The particular vPLC.R1 endpoint 1302 may augment the request with the vPLC ID of vPLC.R1. The control plane 1350 may work with an identity service to check the user's proper credentials for vPLC.R1 (e.g., the user's username and customer C1's tenancy ID). Resource manager 1352 can then identify all resources assigned to vPLC.R1, e.g., server 1320 and hypervisor 1330 based on the vPLC ID and VM 1332a based on the customer tenancy ID for customer C1, and VM instance 1332a is then launched on hypervisor 1330 accordingly.

[0253] 14 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. When a CSP's direct customer registers or subscribes (i.e., signs up) for a cloud service offered by the CSP (i.e., a CSP-provided cloud service), a tenancy or account is created for that customer. Similarly, when a reseller's customer registers or subscribes for a cloud service offered by the individual reseller (i.e., a reseller-provided cloud service), a tenancy or account is created for that customer.

[0254] FIG. 14 illustrates the sign-up process for a particular customer C1 of reseller R1. However, this sign-up process is also applicable to customers of individual resellers. In FIG. 14, in step S1, reseller R1's customer C1 1401 may sign up using a sign-up portal on the console of vPLC.R1 1402 to create a new account by entering user ID credentials (user name or account name (e.g., John)). In step S2, vPLC.R1's console 1402 may enhance the sign-up request by adding a vPLC ID (e.g., vPLC_ID.vPLC.R1) that identifies the vPLC created for reseller R1 to the user credentials and forward the sign-up request to a CSP (control plane (CP) or management plane (MP)) 1404. In some embodiments, the account creation for reseller R1's customer C1 may be performed by reseller R1 with assistance from the CSP's CP or MP.

[0255] Then, in step S3, the CSP creates a customer tenancy (or account T.R1.C1) for reseller R1's customer C1 1401. The newly created tenancy (T.R1.C1) for customer C1 is assigned a tenancy identifier (T_ID.T.R1.C1). A vPLC ID (e.g., vPLC_ID.vPLC.R1) identifying vPLC.R1 for reseller R1 is associated with the newly created customer tenancy ID (e.g., T_ID.T.R1.C1) for customer C1 (shown in 1430) and may be stored in database 1420 along with other authentication information (e.g., account name and password). Other customers of reseller R1 and reseller R2 may also be stored in database 1420. For example, as shown at 1432, the customer tenancy ID of reseller R1's customer C2 (i.e., R1.C2) is associated with the vPLC ID of vPLC.R1 (T_ID.T.R1.C2 and vPLC_ID.vPLC.R1). Also, as shown at 1434, the customer tenancy ID of reseller R2's customer C2 (i.e., R2.C1) is associated with the vPLC ID of vPLC.R2 (T_ID.T.R2.C1 and vPLC_ID.vPLC.R2).

[0256] The association between the vPLC ID and the tenancy IDs of individual resellers' customers can enable isolation between customer tenancies within a vPLC. For example, customer tenancy R1.C1 can be securely isolated from customer tenancy R1.C2 by using different customer tenancy IDs (T_ID.T.R1.C1 for customer C1 and T_ID.T.R1.C2 for customer C2), but both customer tenancies reside in the same vPLC.R1. The tenancies associated with reseller R1 and reseller R2 can be securely isolated by using their respective vPLC IDs (vPLC_ID.vPLC.R1 for reseller R1 and vPLC_ID.vPLC.R2 for reseller R2).

[0257] In step S4, CSP 1404 may return a handle including the customer tenancy ID (e.g., T_ID.T.R1.C1), a temporary default password, and an account name to vPLC.R1 console 1402. In step S5, the vPLC.R1 console may receive the temporary authentication information and forward the customer tenancy ID and password to reseller R1's customer C1 1401. As shown in FIG. 14, reseller R1's customer C1 1401 can sign up for and register for one or more reseller-supplied cloud services by interacting directly with reseller R1 through vPLC.R1 console 1402. Similarly, a CSP's customer may register for a CSP-supplied cloud service by separate interaction and sign-up with the CSP at the CSP's console (not shown).

[0258] As described above with respect to creating a tenancy, a CSP's database (e.g., 1420 in FIG. 14) may store vPLC IDs, customer tenancy IDs, and their corresponding authentication information. In some embodiments, multiple customer tenancy IDs may be associated with the same vPLC ID, as shown in 1430 (T_ID.T.R1.C1 is associated with vPLC_ID.vPLC.R1) and 1432 (T_ID.T.R1.C2 is associated with vPLC_ID.vPLC.R1). Tenancies may be defined at the realm level and thus may have a realm identifier associated with a vPLC.

[0259] 15 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to one embodiment. In FIG. 15, steps S1 and S2 include a login phase after an individual reseller customer completes the sign-up process described in FIG. 14.

[0260] In step S1, a user 1501 associated with a customer of reseller R1 can log in with a customer tenancy ID (or CT ID), username, and password at the vPLC.R1 endpoint 1510. In S2, the user 1501 and their customer tenancy may be verified at the vPLC.R1 endpoint 1510 by going through a verification process, including, but not limited to, user authentication (e.g., verifying valid credentials such as username and password), session management (e.g., session tokens or timeouts), and access control enforcement (based on access privileges, permissions, and policies).

[0261] In S3, the vPLC.R1 endpoint 1510 may return a session handle. The session handle may represent a token or unique identifier returned to a customer after successfully logging into their account. The session handle is used for subsequent requests to access cloud services and resources (e.g., launch an instance) without requiring re-authentication for each individual request during an active login session (or API session).

[0262] When resources are created, a cloud resource ID (CID) may be generated for those resources, which includes the customer tenancy ID. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, ACLs, and load balancers are considered examples of cloud resources and can therefore be referenced by their respective CIDs.

[0263] 16 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. A resource manager (RM) manages the resources of a CSP-provided cloud infrastructure in a region and the resource partitions assigned to each vPLC.

[0264] As described above, a user associated with a customer of a reseller may use a vPLC-specific endpoint to access resources associated with the vPLC and perform one or more functions or operations involving resources allocated to the vPLC. In Figure 16, in step S1, a user 1601 associated with a customer C1 of reseller R1 may launch a compute instance (e.g., a VM instance with 1 GB of RAM) by issuing a request through a vPLC-specific endpoint, vPLC.R1 endpoint 1602.

[0265] In step S2, the vPLC.R1 endpoint 1602 may extend (or annotate) the request by tagging it with the vPLC.R1 endpoint's corresponding vPLC ID (e.g., vPLC_ID.vPLC.R1) to indicate that the request originates from a customer of reseller R1, and forwards the request to resource manager (RM) 1604. In other words, the request, once authenticated by the annotation information (e.g., vPLC ID), can be processed by all policies associated with vPLC.R1. In some embodiments, policies associated with a vPLC may include, for example, the type of resource, the amount of resource that can be allocated (i.e., capacity limits), permissions, etc.

[0266] In step S3, the RM 1604 determines whether the requested resources are available to vPLC.R1 under the policy by querying its database (i.e., resource database) 1606. Based on this query, in step S4, after confirming that the requested resources are available under the policy, the RM may perform resource allocation. In other words, the RM allocates a portion of the resources allocated to vPLC.R1 for reseller R1 to a customer tenancy (T.R1.C1) associated with reseller R1's customer C1 (based on the customer tenancy ID at user login). This resource allocation may be performed by allocating or designating a portion of the available cloud resources (allocated to vPLC.R1) to a user associated with customer C1 based on a policy agreed upon between customer C1 and reseller R1, such as capacity limits or permissions. For example, reseller R1 and its customer C1 may have a contract policy that allows for the use of up to 100 GB of memory. Currently, 80 GB is allocated (or used). If a user associated with customer C1 requests 10 GB, the RM can identify the 10 GB CID provisioned by the CP. If a user associated with customer C1 requests 30 GB, this exceeds the total 100 GB limit, and the request can be rejected.

[0267] In step S5, the RM 1604 retrieves from the resource database a cloud resource ID assigned to a resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S6, the RM passes the resource ID to the vPLC.R1 endpoint 1602. In step S7, the vPLC.R1 endpoint 1602 requests a resource from the control plane (CP) 1608 for a resource having a particular resource ID assigned to the resource allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In other words, the resource manager 1604 can augment the user's request (e.g., step S6) with information associated with the vPLC (e.g., the resource ID assigned to vPLC.R1.C1) that the control plane 1608 can use to provision the resource (e.g., step S8).

[0268] In step S8, the control plane may provision the requested resources (e.g., launch a specific VM instance and storage). The resources may include compute, networking, storage, etc. In step S9, the control plane 1608 may respond to the vPLC.R1 endpoint 1602 with a status and a resource handle. The resource handle may represent a unique resource identifier assigned to the provisioned resource (e.g., a cloud resource ID assigned to the provisioned compute instance (e.g., VM), storage volume, virtual cloud network (VCN or subnet), database, etc.). The resource handle may enable the customer to access and manage the provisioned resource through an API or command line tools.

[0269] In step S10, the status and resource handle are forwarded to user 1601 associated with customer C1 of reseller R1. In step S11, the control plane may notify the RM regarding the resource provisioning status (e.g., success or failure). In step S12, the RM may update a record in resource DB 1606 that tracks resource usage for customer C1 associated with the user. Continuing with the above example, the record may be updated to indicate that the total amount of memory used by customer C1 is 90 GB after successfully provisioning 10 GB in addition to the previously provisioned 80 GB, or that the total remains at 80 GB after the user's request was denied. Records maintained by the resource DB may be useful for enforcing policies associated with vPLCs.

[0270] Figure 17 is a flow diagram illustrating a vPLC resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. Figure 17 is an alternative embodiment to Figure 16. Instead of using a Resource Manager (RM) as a central role for communicating with all subsystems and receiving requests to coordinate the resource allocation and provisioning process, a Control Plane (CP) may also be used for this central role.

[0271] For example, in FIG. 17 , in step S1, a user 1701 associated with a customer C1 of reseller R1 may issue a request to a vPLC-specific endpoint, vPLC.R1 endpoint 1702. In step S2, vPLC.R1 endpoint 1702 may extend the request by tagging it with a corresponding vPLC ID to indicate that the request originates from a customer of reseller R1, and forwards the request to control plane (CP) 1704. In step S3, CP 1704 forwards the request to resource manager (RM) 1706. In step S4, RM 1706 determines whether the requested resources are available for vPLC.R1 by querying its database (i.e., resource database) 1708. Based on this query, in step S5, if the requested resources are available, the RM performs resource allocation.

[0272] In step S6, the RM retrieves from the resource database the resource ID assigned to the resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S7, the RM passes the resource ID to the CP 1704. In step S8, the control plane may provision the requested resource. In step S9, the control plane 1704 may respond to the vPLC.R1 endpoint 1702 with a status and a resource handle. In step S10, the status and resource handle may be forwarded to the user 1701. In step S11, the control plane may notify the RM regarding the provisioning status of the resource. In step S12, the RM may update a record in the resource DB that tracks resource usage for the customer associated with the user.

[0273] 17, the CP 1704 may play a central role by forwarding the resource request to the RM 1706 (e.g., step S3), receiving the resource ID from the RM (e.g., step S6), and provisioning the requested resources (e.g., step S8). Compared to FIG. 16, this embodiment avoids the need for the vPLC.R1 endpoint 1702 to perform extra steps, such as requesting resources from the CP as shown in steps S7 and S8 of FIG. 16.

[0274] FIG. 18 is a simplified diagram illustrating a use case in which a vPLC is hosted by the infrastructure of a first realm while associated with a second, different realm, according to one embodiment. In FIG. 18, there are two separate realms (CSP-provided realm A 1802 and realm B 1812). Each realm is a logical collection of regions. Realms are isolated from each other and do not share data. Thus, realm A 1802 is isolated from realm B 1812, and there is no data sharing or communication between the two realms. A tenancy created for a CSP customer exists only in one realm and cannot be used to access another realm or region in another realm. Realms span regions within that realm, allowing the CSP to provide a defined level of service that meets the needs of a particular organization. For example, for a high level of security and isolation, a CSP may create realms, such as a “Government” realm created for government entities that are customers of the CSP. A CSP may also create a second realm with a lower level of security and isolation, such as a "Commercial" realm for the CSP's non-government commercial customers. Examples of realms a CSP may create are (a) a Commercial realm, (b) a US Government FedRAMP Authorized realm, (c) a US Government IL5 Authorized realm, (d) a Country A Government realm, (e) a Country B Government realm, etc. For example, in FIG. 18, realm A 1802 may have a higher security posture than realm B 1812.

[0275] A realm can have one or more regions. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). This identity and trust profile information is shared between regions in the same realm, so regions within a realm can communicate with each other. The realm's identity and trust profile are configured to prevent inter-realm communication.

[0276] In the example shown in Figure 18, realm A 1802 includes region A 1804. The infrastructure provided by the CSP in region A 1804 is organized as datacenter 1806. In Figure 18, datacenter 1806 is shown as a dedicated regional cloud datacenter (DRCC) hosted on customer premises. A DRCC may represent a fully managed cloud region built with CSP-designed high-performance infrastructure to help customers bring cloud primitives and services closer to their existing data and applications. A DRCC may be dedicated to a single customer and operate within the customer's datacenter while being fully managed by the CSP.

[0277] In FIG. 18 , realm B 1812 includes region B 1814. The infrastructure provided by the CSP in region B 1814 is organized as data center 1816. As shown in FIG. 18 , data center 1816 hosts vPLC 1818. Thus, a portion of the infrastructure provided by the CSP in data center 1816 of region B 1812 is allocated to vPLC 1818. Thus, vPLC 1818 is physically located in region B 1814 of realm B 1812.

[0278] As previously mentioned, when a vPLC is created, it is associated with a particular realm as part of its configuration. Typically, a vPLC is associated with the same realm associated with the region in which the CSP-provided regional infrastructure hosting the vPLC is located. In some embodiments, when a vPLC is configured, it may be associated with a realm different from the realm associated with the regional infrastructure hosting the vPLC. For example, in the example shown in FIG. 18 , vPLC 1818 may be associated with Realm A 1802 rather than Realm B 1812. The dashed box 1820 and associated arrow 1830 are meant to indicate that vPLC 1818 is physically hosted by the data center 1816 of Realm B 1812, but is logically or virtually associated with and belongs to Realm A 1802. Realm B 1812 is referred to as the hosting realm of vPLC 1818. Realm A 1802 is referred to as the logical or virtual realm of the vPLC 1818.

[0279] As a result, vPLC 1818 and datacenter 1806 exist in the same realm, i.e., realm A 1802. Because infrastructure in the same realm shares the same identity and trust profile, vPLC 1818 is configured with the identity and trust profile of realm A 1802. Configuring the identity and trust profile information of realm A 1802 can be achieved by creating a mapping in vPLC 1818. For example, the identity service API of the virtual realm (i.e., realm A 1802) can call the vPLC of the host realm (i.e., realm B 1812), and the host realm can accept the call and establish trust between the two realms (i.e., share identity and trust profile information). When an instance is created in a customer tenancy of the vPLC of the host realm, the resource ID and tenancy ID associated with the newly created instance in the host realm can be mapped to a new resource ID and tenancy ID in the virtual realm. Thus, a table of mappings (or "shadow tenancy") may be maintained in the host realm (i.e., realm B 1812). The same process and mapping occurs in the virtual realm (i.e., realm A 1802). Thus, both vPLC 1818 and realm A 1802 (i.e., the virtual realm) share the same identity and trust profile information. As a result, even though vPLC 1818 is physically hosted by a different realm B 1812 infrastructure, vPLC 1818 and datacenter 1806 can communicate with each other because they reside in realm A 1802. vPLC 1818 may not be able to communicate with other infrastructure in region B 1814 associated with realm B 1812.

[0280] Additionally, vPLC 1818 is physically hosted by region B 1814 but is logically associated with region A 1804. Region B 1814 is referred to as the hosting region or host region of vPLC 1818. Region A 1804 of realm A 1802 is referred to as the logical region or virtual region of vPLC 1818. When a vPLC is created, a virtual realm and virtual region for the vPLC may be configured.

[0281] As described above, a resource ID for a resource includes a portion that identifies the realm associated with the resource and a portion that identifies the region associated with the resource. A vPLC is considered a resource and is therefore assigned a resource ID. In one embodiment, a vPLC ID, which uniquely identifies a vPLC, is a type of resource ID. A vPLC ID includes a portion that identifies the virtual realm associated with the vPLC identified by the vPLC ID and a portion that identifies the virtual region associated with the vPLC. Thus, for purposes of identity management in Realm A 1802, a vPLC is considered part of the realm and region identified by the vPLC ID corresponding to the vPLC.

[0282] In one embodiment, when a vPLC is hosted by a region and a realm but associated with a different virtual region and a different virtual realm, the configuration of the vPLC as described above may be performed using Virtual Bootstrap Environment (ViBE) technology. A ViBE may represent a virtual cloud network (VCN) provisioned in an overlay of an existing region (e.g., a "host region"). The provisioned ViBE is connected to the new region using a communication channel (e.g., an IPSec Tunnel VPN). Certain essential core services (or "seed" services), such as a deployment orchestrator and public key infrastructure (PKI) services, may be provisioned in the ViBE. These services can provide the functionality necessary for bringing hardware online, establishing a chain of trust to the new region, and deploying other services in the new region. The use of a Virtual Bootstrap Environment prevents circular dependencies between bootstrap resources by utilizing resources from the host region. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0283] For example, in some embodiments, Realm B 1812 may create a ViBE within its realm, which shares identity and trust profile information with Realm A 1802. The ViBE can then be used in Realm B 1812 to bootstrap new regions in Realm A 1802. Additional details related to ViBEs are described in the following applications, the contents of all of which are incorporated herein by reference:

[0284] (1) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "VIRTUAL BOOTSTRAP ENVIRONMENT FOR BUILDING REGIONAL DATA CENTERS" (2) U.S. Patent Application No. 18 / 105,768, filed February 3, 2023, entitled "TECHNIQUES FOR A VIRTUAL BOOTSTRAP ENVIRONMENT IN A DISTRIBUTED VIRTUAL PRIVATE NETWORK" (3) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "TECHNIQUES FOR MIGRATING SERVICES FROM A VIRTUAL BOOTSTRAP ENVIRONMENT" In the architecture shown in FIG. 18 , a vPLC is hosted by a region and a realm, but is associated with a different virtual region and a different virtual realm, which can be used for various purposes. For example, in the example shown in FIG. 18 , vPLC 1818 may function as a backup for data center 1806. This allows for disaster recovery (DR) backups in various regions. For example, if data center 1806 is a DRCC located on a customer's premises, the customer may request a CSP to provide backup services for DRCC 1806. In response, the CSP may select vPLC 1818 as a backup for DRCC 1806. vPLC 1818 is physically hosted in region B 1812, which is in a different region away from region A 1802, which hosts DRCC 1806. This may be a cost-effective solution because the CSP does not need to build a separate data center for the customer of realm A 1802 just for backup purposes. In some embodiments, the customer may not even be aware of where the backup is being performed, i.e., the presence and location of the vPLC 1818 may be transparent to the customer. The vPLC used for backup may be physically hosted in a CSP-provided cloud in a different realm and region.

[0285] Since the purpose of a DR site is to minimize the occurrence of correlated failures by introducing geographic separation between the primary and DR sites and avoiding simultaneous changes to both the primary and secondary locations, the DRCC 1806 can enjoy DR guarantees if the CSP can ensure that the DRCC and its paired vPLC 1818 (and therefore its hosting region in Realm B 1812) do not change simultaneously.

[0286] 19 is a flowchart 1900 illustrating an exemplary method for creating a vPLC hosted by a CSP provisioning infrastructure in a particular region in a particular realm, while virtually associated with a different region in a different realm, according to one embodiment. The process illustrated in FIG. 19 may be performed as part of the process performed for creating the vPLC. In one embodiment, the process illustrated in FIG. 19 may be performed primarily by the CSP provisioning infrastructure in the hosting region and realm.

[0287] Processing begins at 1902 with the receipt of information identifying a host region of a host realm in which a vPLC will be physically hosted. For example, in the embodiment shown in Figure 18, information may be received that a vPLC will be created in region B 1814 of realm A 1812. The region and realm identified at 1902 represent the host region and host realm of the vPLC to be created.

[0288] At 1904, information may be received that identifies a region of a particular realm. The particular realm is different from the host realm identified at 1902. Furthermore, the region and particular realm identified at 1904 will be associated as a virtual region and virtual realm of the vPLC being created. For example, in the embodiment shown in FIG. 18, information may be received that the vPLC is to be virtually associated with region A 1804 of realm A 1802.

[0289] At 1906, identity and trust profile information is obtained for the particular realm identified at 1904. The information obtained at 1906 may include identity authentication information (including passwords, certificates, etc.) associated with the realm that will be the virtual realm of the vPLC.

[0290] At 1908, a vPLC ID is generated for the vPLC being created. The vPLC ID identifies the region and particular realm identified in the information received at 1904.

[0291] At 1910, a vPLC is created in the host realm and resources are allocated to the vPLC. The allocated resources are selected from resources in the infrastructure provided by the CSP in the region and realm identified in 1902. These resources may include one or more physical resources (e.g., racks, servers, routers, memory, or storage resources) and / or logical resources (e.g., hypervisors, virtual machines, virtual routers). For example, in the embodiment shown in FIG. 18, resources are allocated to the vPLC from a data center 1816.

[0292] At 1912, a vPLC is configured using the identity and trust profile information obtained at 1906. For example, in the embodiment shown in Figure 18, the vPLC may be configured using the identity and trust profile information for Realm A 1802.

[0293] At 1914, the vPLC initiates communication with the CSP-provided infrastructure in the particular realm (the vPLC's virtual realm) identified at 1904. For example, in the embodiment shown in Figure 18, the vPLC 1818 communicates with the data center 1806. This communication may be initiated by the vPLC 1818 or by the data center 1806.

[0294] Virtual Private Label Cloud (vPLC) - Console Customization The techniques described in this disclosure may enable multiple console experiences through different consoles for different vPLCs operating in the same physical region. A CSP may provide a console for each reseller, including a console UI and a console server, to manage the reseller's vPLC resources and customize the console UI. For purposes of this application, a CSP console may represent a set of graphical user interfaces (GUIs) provided by the CSP. Users associated with the CSP's customers can interact with the infrastructure and CSP-supplied cloud services. For example, a console may be implemented as a web application including a set of one or more GUIs (e.g., a set of web pages). A vPLC-specific console may represent a set of GUIs provided by the CSP, which may be customized by the reseller associated with the vPLC. Users associated with the reseller's customers can interact with the infrastructure and reseller-supplied cloud services. The console may be executed by one or more console servers.

[0295] In some embodiments, the techniques described herein can generate multiple consoles, each associated with one or more endpoints, which may involve multiple IP addresses served by the same underlying infrastructure. For example, in some embodiments, a single console server may run multiple consoles for multiple vPLCs and CSPs. This model is referred to herein as a shared console model. In other embodiments, a single console server may be dedicated to vPLC-specific consoles. This model is referred to herein as a dedicated console model. In some embodiments, a hybrid model combining the shared console model and the dedicated console model may be possible. The backend console server may use each user's login information in conjunction with the console's extended vPLC-related information to determine who is accessing which console and respond with a customized console experience for each user. This technique for vPLC-specific console customization allows for the sharing of backend resources and reduced costs, while providing different console experiences customized by different resellers.

[0296] The disclosed technology may also enable console customization for vPLC-specific consoles. This customization not only allows users associated with a reseller's customers to visualize cloud resources associated with the reseller's vPLC, but also allows the reseller to add additional aspects (e.g., look and feel) to each user's experience. In other words, users associated with a CSP's customers may see a default view when they log in to a standard CSP console. Users of the reseller's customers will have a different console experience when they log in through a vPLC-specific console.

[0297] Console customizations for a vPLC-specific console may be a set of console UI customizations for an individual reseller. For example, customizations may include, but are not limited to, interactivity, colors, cascading style sheets (CSS), HTML pages, and how information is displayed (e.g., endpoint URLs for specific services). Customizations may be based on console policies, which are determined in part by the CSP-reseller agreement. Console policies (or user experience policies) may represent specific information, messages, or graphical user interfaces that a reseller does or does not want displayed on the console UI to its customers and associated users.

[0298] In some embodiments, the customized set of console UI may include the reseller's authentication / authorization mechanisms, APIs, plug-ins, and links to other advertisements or pages. The CSP may provide the reseller with a software development kit (SDK) that invokes the reseller's own plug-ins. Thus, when a user clicks a link in the console, the user can invoke a plug-in from the CSP or the reseller.

[0299] In some embodiments, the customized UI in the vPLC-specific console may include one or more URLs corresponding to one or more endpoints created for the vPLC. A user can select an endpoint by selecting a URL, which may execute a cloud service function. The endpoints included in the vPLC-specific console may vary depending on the services offered to the reseller's customers based on the reseller's subscription as part of the console customization. Illustratively, a CSP may make available services S1, S2, and S3. If a CSP reseller R1 only subscribes to S1 and S2, only the service endpoints associated with S1 and S2 may be presented in the vPLC.R1-specific console as uniform resource locators (URLs) for the user to select and use.

[0300] Finally, in some embodiments, this multi-console experience may be achieved by allowing a console's user interface (UI) to add a vPLC context enabled by a vPLC-aware login session. In other words, a user associated with a customer of reseller R1 can be authenticated using the tenancy and credentials stored in that login session. For example, in one embodiment, such a vPLC context may include a URL associated with the console, and instead of using the CSP's name (e.g., oraclecloud.com), the URL may be formatted with the prefix "console" and a suffix containing the reseller's name, with additional region information (e.g., us-phoenix-1) added between the two. As an example, if a user logs into a vPLC.R1-specific console that may have an associated URL "console.us-phoenix-1.vplcR1Cloud.com," this URL may point to the same CSP management backend server (located in physical region us-phoenix-1) that also serves a second reseller R2's console that may have the URL "console.us-phoenix-1.vplcR2cloud.com." In other embodiments, both "console.us-phoenix-1.vplcR1Cloud.com" and "console.us-phoenix-1.vplcR2cloud.com" may be running on different servers in the same physical region (us-phoenix-1).

[0301] In some embodiments, a reseller may take advantage of a CSP-provided infrastructure in a region by building its own console that users associated with the reseller's customers can use to access physical resources provided by the CSP under the reseller's established terms and user experience policies (or console policies). For example, a reseller may customize a customizable console implementation provided by a CSP by modifying the cascading style sheets (CSS) and logo to match the reseller's trademarks and preferred user experience. In some embodiments, a reseller may implement an entire console website itself. If a reseller implements its own console, it may choose to host the website outside of the region in which the CSP provides vPLC functionality.

[0302] In summary, based on the vPLC settings configured by the reseller, users can be provided with a customized console experience (e.g., UI, colors, CSS stylesheets, HTML pages, or service endpoint URLs) specific to a particular vPLC console. For example, in a shared console model, a German customer of a reseller who focuses on HPC may log in to a vPLC-specific console that highlights HPC products and articles. Conversely, a French customer of another reseller who focuses on education workloads may log in to a vPLC-specific console that highlights Coursework Cloud applications and features. Although both vPLC-specific consoles share (or are run by) the same console server, the users can have very different console experiences.

[0303] The techniques described in this disclosure may enable different console models: a dedicated console model (i.e., different console servers for consoles of different vPLCs), a shared console model (the same console server providing a different console experience for each supported vPLC), and a hybrid console model (a combination of dedicated and shared console models).

[0304] FIG. 20 illustrates a dedicated vPLC console model, according to one embodiment. The distributed environment 2000 illustrated in FIG. 20 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment 2000 may include more or fewer systems or components than those illustrated in FIG. 20, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 20 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0305] In FIG. 20, there are three consoles, each associated with an endpoint and a set of UIs. For example, CSP console 2002 has a set of UIs 2002b and is associated with a set of endpoints 2002a. VPLC.R1 console 2004 has a set of UIs 2004b and is associated with a set of endpoints 2004a. VPLC.R2 console 2006 has a set of UIs 2006b and is associated with a set of endpoints 2006a. Each console may be implemented by a dedicated console server (e.g., CSP console server 2012 for CSP console 2002, vPLC.R1 console server 2014 for vPLC.R1 console 2004, and vPLC.R2 console server 2016 for vPLC.R2 console 2006). These console servers reside in the same CSP-provided infrastructure in a given region. A shared control plane (CP) 2030 may be used to manage all the console servers. For example, if user 2082 logs into vPLC.R1 console 2004, the URL associated with this console (console.vplcR1cloud.com) can direct the user's requests directly to vPLC.R1 console server 2014.

[0306] As previously described, the techniques described in this disclosure may enable each reseller to manage its console server and customize the console experience for its customers. Console customization may be performed through the CSP console and CP. For example, in FIG. 20 , reseller R1 2086 may customize a set of console UIs for the vPLC.R1 console 2004 by logging into the CSP console 2002, which may pass customization requests and information to the vPLC.R1 console server 2014 through the CP 2030. The customized console UIs may be stored in the namespace 2022 associated with the vPLC.R1 console server 2014. Similar console customization may be performed for reseller R2 2088 through the CSP console 2002 and CP, with R2's customization information stored in the namespace 2024 associated with the vPLC.R2 console server 2016. In some embodiments, the namespace 2022 associated with vPLC.R1 and the namespace 2024 associated with vPLC.R2 may reside in the same database or in different databases. The process for configuring console customization is discussed in more detail below.

[0307] After console customization is configured, when a user of the reseller's customer logs into the console, the console server may generate a console session specific to the vPLC associated with the console. A URL included in the console session (e.g., "console.vplcResellerCloud.com") may direct the user to a dedicated console server that may load the reseller's customized console experience.

[0308] FIG. 21 illustrates a shared vPLC console model, according to one embodiment. The distributed environment 2100 illustrated in FIG. 21 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment 2100 may include more or fewer systems or components than those illustrated in FIG. 21, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 21 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0309] In a shared console model, the console server / infrastructure may be shared across multiple vPLCs, with the UI logic understanding how to manage multiple vPLC views. The shared console server may then bootstrap to add a new namespace for each new vPLC via API calls to the console service in the vPLC control plane. There may still be multiple vPLC-specific console URLs, each with a unique domain name associated with the console. However, these console URLs may point to the same backend server (i.e., console server). Based on the login information (e.g., tenancy) provided by the reseller's customer's users at each console, the backend console server can determine the specific vPLC console experience to present to the user.

[0310] 21, there are three consoles with associated endpoints and sets of UIs. For example, CSP console 2102 has a set of UIs 2102b and is associated with a set of endpoints 2102a. VPLC.R1 console 2104 has a set of UIs 2104b and is associated with a set of endpoints 2104a. VPLC.R2 console 2106 has a set of UIs 2106b and is associated with a set of endpoints 2106a. The vPLC-specific consoles (2104 and 2106) may each have a unique domain name (e.g., vplcR1Cloud.com and vplcR2Cloud.com) while sharing (or being executed by) console server 2112. A shared control plane (CP) 2130 may be used to control and create the vPLC namespaces (or partitions) in a shared database 2140 associated with backend console server 2112. Each vPLC has a corresponding namespace (or partition) in the shared database (eg, namespace 0 2120 for CSP, namespace 1 2122 for vPLC.R1, and namespace 2 2124 for vPLC.R2).

[0311] As previously described, the techniques described herein may enable each reseller to manage its console server and customize the console experience for its customers. Console customization may be performed through the CSP console and CP. For example, in FIG. 21 , reseller R1 2186 may customize the console display of vPLC.R1 console 2104 by logging into CSP console 2102, which may pass customization requests and information to shared console server 2112 through CP 2130. The customized console UI using the customization information may be stored in namespace 1 2122 associated with reseller R1's vPLC ID in database 2140. Similar console customization may be performed for reseller R2 2188 through the CSP console 2110 and CP, and the customized console UI may be stored in namespace 2 2124 associated with reseller R2's vPLC ID in database 2124 using the customization information provided by R2. The process for configuring console customization is discussed in more detail below.

[0312] After the console customization is configured, when a user of the reseller's customer logs in to a particular vPLC console (e.g., vPLC.R1 console 2104), the user's authentication information (including tenancy) and related vPLC information (e.g., vPLC ID) may be forwarded to the shared console server 2112, which may then route the user's request to the appropriate vPLC namespace (e.g., vPLC.R1 namespace1 2122) based on the related vPLC information and retrieve the customized console UI according to the console implementation policies of that vPLC.

[0313] FIG. 22 illustrates a hybrid vPLC console model, according to one embodiment. Sharing of a console server may refer to sharing of a console server by at least two consoles. In one embodiment, a subset of the consoles may share (or be run by) a console server, another subset of the consoles may share a different console server, while a third subset of the consoles may have a dedicated console server. For example, in FIG. 22, a first subset of the consoles (CSP console 2202 for the CSP's direct customers and vPLC.R1 console 2204 for reseller R1) may share console server 2232. Also, a second subset of the consoles (vPLC.R3 console 2212 for reseller R3 and vPLC.R4 console 2214 for reseller R4) may share console server 2236. Finally, a third subset of the consoles (vPLC.R2 console 2206 for reseller R2) may have a dedicated console server 2234. The shared consoles 2232 and 2236 may each follow the shared vPLC console model described above with respect to Figure 21. The dedicated console 2234 may follow the dedicated vPLC console model described above with respect to Figure 20.

[0314] In some embodiments, the determination of a shared console server for a subset of consoles may depend on an agreement between the requesting reseller and the CSP, as well as resource availability and capacity. For example, if three consoles (e.g., CSP.C1, vPLC.R1, and vPLC.R3) share a console server and a fourth reseller, R4, requests a shared console, the CSP and reseller may have an agreement to distribute the resource load by separating the four consoles into two subsets of consoles, each sharing a console server, as shown in Figure 22. For example, a first subset may include CSP.C1 consoles 2202 and vPLC.R1 consoles 2204, and a second subset may include vPLC.R3 consoles 2212 and vPLC.R4 consoles 2214.

[0315] FIG. 23 is a flowchart illustrating a process for configuring vPLC console customization, according to one embodiment. The process illustrated in FIG. 23 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 23 and described below is exemplary and not intended to be limiting. While FIG. 23 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 23 may include more or fewer steps than those depicted in FIG. 23.

[0316] For example, in one embodiment, the process illustrated in Figure 23 may be performed by a CSP, and some steps may be performed by components within the CSP-provided regional infrastructure (e.g., control plane (CP), resource manager (RM), and management plane (MP)), or by one or more CSP-provided cloud services.

[0317] In step 2302 of Figure 23, a CSP may receive a request to create a console for a vPLC associated with a reseller, and the request is sent to the CSP CP for execution of such request. In some embodiments, the request may be initiated by a reseller (e.g., reseller R1 2086 of Figure 20 or 2186 of Figure 21) through a CSP console (e.g., 2002 of Figure 20 or 2102 of Figure 21). In some embodiments, the request may originate from a CSP-provided infrastructure that performs provisioning of a vPLC associated with the reseller in a region, as described above with respect to Figure 5.

[0318] In step 2304, the CSP CP may receive information indicating the type of console to be created (whether the reseller is requesting a dedicated console server or a shared console server for the vPLC console to be created). In step 2306, the configuration process may determine whether the request is for a dedicated console server or a shared console server. If the request is for a dedicated console server, the process may proceed to step 2312. If the request is for a shared console server, the process may proceed to step 2308. Both types of consoles may have common steps 2350-2358.

[0319] If the request is for a dedicated console server, then in step 2312, the CSP may create a dedicated console server (e.g., 2014 in FIG. 20) to serve the console (e.g., 2004 in FIG. 20) created for the vPLC identified in 2302. In step 2314, the dedicated console server created in 2312 may be associated with the namespace (e.g., 2202 in FIG. 20) of the vPLC for which the console is being created.

[0320] In response, at step 2350, a console may be generated that includes a set of UIs customized for the reseller corresponding to the vPLC. For example, in FIG. 20, vPLC.R1 console 2004 is generated with a set of customized UIs 2004b. At step 2352, the set of UIs customized using the customization information and associated with the console may be stored in a namespace associated with the vPLC. For example, in FIG. 20, the set of UIs customized for vPLC.R1 is stored in namespace 1 2022.

[0321] In step 2354, a set of endpoints is also created for the vPLC. For example, in FIG. 20, a set of endpoints 2004a is created for vPLC.R1. Based on the services for which the reseller has subscribed, the CSP may determine the service endpoints to be created for the vPLC. In step 2356, the console created in 2350 and the set of endpoints created in 2354 are associated together. For example, in FIG. 20, the set of endpoints 2004a is associated with the vPLC.R1 console 2004. Finally, in step 2358, access to the console server associated with the vPLC and the set of endpoints created for the vPLC may be provided. For example, in FIG. 20, the CSP may provide the reseller, the reseller's customers, and their associated users (e.g., 2082 in FIG. 20) with access to the console server 2014 associated with vPLC.R1 and the set of endpoints 2004a created for vPLC.R1.

[0322] In a shared console model, when a reseller requests a console for a vPLC, the vPLC ID associated with the vPLC may be passed to the CP2130 and the shared console server 2112, and settings such as identification of the corresponding namespace associated with the vPLC ID may be performed. In contrast, in a dedicated console model, the CP2030 similarly receives the vPLC ID associated with the vPLC, but the vPLC ID may not be used.

[0323] Referring again to step 2306, if the request is for a shared console server, then in step 2308 the CSP may check whether a shared console server already exists to serve the console being created. If a shared console server exists in step 2309, the process proceeds to step 2340, where the existing shared console server may be associated with the namespace of the vPLC for which the console is being created. For example, in Figure 21, existing console server 2112 may be associated with namespace 1 2122 of vPLC.R1.

[0324] If a shared server does not exist, the process proceeds to step 2342. In step 2342, the CSP may create a shared console server. In step 2344, the newly created shared console server may be associated with the namespace of the vPLC for which the console is being created.

[0325] Steps 2350-2358 are similar for the dedicated console model and the shared console model. In step 2350, a console may be generated that includes a set of UIs customized for the reseller corresponding to the vPLC. For example, in FIG. 21, a vPLC.R1 console 2104 having a set of customized UIs 2104b is generated. In step 2352, the set of UIs customized using the customization information and associated with the console may be stored in a namespace associated with the vPLC. For example, in FIG. 21, the set of UIs customized for vPLC.R1 is stored in namespace 1 2122.

[0326] In step 2354, a set of endpoints is also generated for the vPLC. For example, in FIG. 21, a set of endpoints 2104a is generated for vPLC.R1. Based on the services for which the reseller has subscribed, the CSP may determine the service endpoints to be generated for the vPLC. In step 2356, the console generated in 2350 and the set of endpoints generated in 2354 are associated together. For example, in FIG. 21, a set of endpoints 2104a is associated with the vPLC.R1 console 2104. Finally, in step 2358, access to the console server associated with the vPLC and the set of endpoints generated for the vPLC may be provided. For example, in FIG. 21, the CSP may provide the reseller, the reseller's customers, and their associated users (e.g., 2182 in FIG. 21) access to the shared console server 2112 associated with vPLC.R1 and the set of endpoints 2104a generated for vPLC.R1.

[0327] FIG. 24 is a flowchart illustrating a login process for a user associated with a customer of a reseller associated with a vPLC using a vPLC-specific console, according to one embodiment. The process illustrated in FIG. 24 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 24 and described below is exemplary and not intended to be limiting. While FIG. 24 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 24 may include more or fewer steps than those depicted in FIG. 24.

[0328] For example, in one embodiment, the process illustrated in Figure 24 may be performed by a CSP, with some steps being performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP)), or by one or more CSP-provided cloud services, such as an identity cloud service.

[0329] At step 2410, a user of the reseller's customer associated with the vPLC may access a vPLC-specific console landing page generated for the vPLC. For example, the landing page may be a login page on the console's GUI. At step 2412, the user may be prompted to enter login information. For example, the user may be required to provide identification and authentication information (e.g., username and password).

[0330] In step 2414, login information may be received from the user. In step 2416, the login information may be validated. For example, the login information provided by the user may be received by an Identity Cloud Service (IDCS), which may be part of the CSP-provided cloud services. The IDCS may perform validation (including authentication, authorization, and checking the user's tenancy) based on the login information. If the login information is deemed invalid in step 2420, the login process may end in step 2424.

[0331] If the login information is deemed valid in step 2420, a login session may be established for the user. For example, a vPLC-specific console may create a login session and add a session ID (i.e., a unique identifier for the duration of the login) and a vPLC ID associated with the particular console to the session information for the login session. The session information may represent information associated with an active session between the user and the identified dedicated console server (and service) to maintain state and provide secure communications.

[0332] FIG. 25 is a flowchart illustrating a method for serving a console GUI to a user associated with a customer of a reseller associated with a vPLC, according to one embodiment. The process illustrated in FIG. 25 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 25 and described below is exemplary and not intended to be limiting. While FIG. 25 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 25 may include more or fewer steps than those depicted in FIG. 25.

[0333] For example, in one embodiment, the process shown in Figure 25 may be performed by any component (of Figure 20 or Figure 21) responsible for facilitating the serving of a GUI to a user. For example, step 2510 may be performed by a vPLC-specific console to receive a request from a user.

[0334] In step 2510, a request to access one of a set of GUIs in a vPC-specific console may be received from a user associated with a customer of a reseller associated with the vPLC. For example, in Figure 21, a user 2182 associated with a customer of reseller R1 associated with vPLC R1 may request one of the reseller-supplied cloud services.

[0335] In step 2512, a vPLC is identified for this request. The vPLC associated with the user may be identified based on login information from the user and vPLC-related information (e.g., a vPLC ID) extended by the vPLC-specific console. In step 2514, a console server serving the vPLC-specific console / a console server running the vPLC-specific console may be identified. Once the vPLC is identified and its console server model (shared console server model or dedicated console server model) is determined, a console server running the vPLC-specific console can be determined. For example, in FIG. 20, the dedicated console server 2014 may be identified for the user logged in to the vPLC.R1 console 2004. In FIG. 21, the shared console server 2112 may be identified for the user logged in to the vPLC.R1 console 2104 based on the vPLC-related information.

[0336] In step 2516, the request may be forwarded to the vPLC-specific console server identified in 2514. In step 2518, the console server receiving the request serves one of a set of GUIs included in the vPLC-specific console in response to the request in 2510. For example, in the case of the shared console server model of FIG. 21, the identified shared console server (e.g., shared console server 2112 of FIG. 21) may use the vPLC ID to access a customized console UI (e.g., HTML pages and endpoints to be displayed in the console) stored in a namespace associated with the vPLC ID in the shared database (e.g., namespace1 2122 of FIG. 21). This customized GUI may be served and displayed to the user. In the case of the dedicated console server model of FIG. 20, the identified dedicated console server (e.g., vPLC.R1 console server 2014 of FIG. 20) may access a customized console UI specific to reseller R1 stored in namespace1 2022. This customized GUI may then be served and displayed to the user.

[0337] Since a user may access multiple GUIs and continuously request the same login session to the vPLC-specific console, the vPLC-specific console may make an API call using session information to the console server (dedicated console server or shared console server) that runs the vPLC-specific console. For this reason, steps 2510 to 2518 may be repeated.

[0338] In one embodiment, the console GUI served to a user may include one of a set of endpoints associated with the vPLC. Multiple GUIs may be served to a user in a particular session. The GUI may include a URL corresponding to the endpoint created for that vPLC. The user may select that endpoint by selecting the URL. In response, a function corresponding to that endpoint may be executed.

[0339] Exemplary CSPI Architecture As mentioned above, infrastructure as a service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also supply various services associated with these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Accordingly, because these services can be policy-driven, IaaS users can maintain application availability and performance by implementing policies that drive load balancing.

[0340] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and use the cloud provider's services to install other elements of their application stack. For example, a user can log in to an IaaS platform to create virtual machines (VMs), install operating systems (OSs) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on those VMs. The customer can then use the provider's services to perform a variety of functions, such as distributing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0341] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but need not be, a third-party service that specializes in providing IaaS (e.g., providing, renting, selling). An entity may also deploy a private cloud and become its own infrastructure service provider.

[0342] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a provisioned application server, etc. It may also include the process of provisioning the server (e.g., installing libraries, daemons, etc.), which is often managed below the hypervisor layer (e.g., server, storage, network hardware, and virtualization) by the cloud provider. As such, the customer may be responsible for handling (e.g., on self-service virtual machines (which can be spun up on demand)), middleware, and / or application deployment, etc.

[0343] In some instances, IaaS provisioning may refer to obtaining computers or virtual hosts for use and even installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, which may need to be performed first.

[0344] IaaS provisioning sometimes presents two distinct challenges. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure after all the provisioning has occurred (e.g., adding new services, modifying services, removing services, etc.). Sometimes these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., the required components and how they interact) can be specified by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., resource dependencies and how they work together) can be described declaratively. Sometimes, once the topology is specified, workflows can be generated to configure and / or manage the various components described in the configuration files.

[0345] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as a core network. Also, in some examples, there may be one or more inbound / outbound traffic group rules provisioned to define how inbound and / or outbound traffic of the network is configured, as well as one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure may evolve incrementally as there is a demand and / or addition of more infrastructure elements.

[0346] In some cases, employing continuous deployment techniques may enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more (but often many) different production environments (e.g., across various geographic locations, possibly across the globe). However, in some examples, the infrastructure into which the code will be deployed is configured first. In some cases, provisioning can be performed manually, provisioning tools can be used to provision resources, and / or deployment tools can be used to deploy the code after the infrastructure has been provisioned.

[0347] 26 is a block diagram 2600 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 2602 may be communicatively coupled to a secure host tenancy 2604, which may include a virtual cloud network (VCN) 2606 and a secure host subnet 2608. In some examples, the service operator 2602 may employ one or more client computing devices, which may be portable handheld devices (e.g., iPhones, mobile phones, iPads, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass head-mounted displays, etc.) running software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and capable of using the Internet, email, short message service (SMS), BlackBerry, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, examples of which include personal computers and / or laptop computers running various versions of the Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may also be workstation computers running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over a network that can access VCN 2606 and / or the Internet.

[0348] VCN 2606 may include a local peering gateway (LPG) 2610, which may be communicatively coupled to a secure shell (SSH) VCN 2612 via an LPG 2610 included in the SSH VCN 2612. The SSH VCN 2612 may include an SSH subnet 2614, which may also be communicatively coupled to a control plane VCN 2616 via an LPG 2610 included in the control plane VCN 2616. The SSH VCN 2612 may also be communicatively coupled to a data plane VCN 2618 via the LPG 2610. The control plane VCN 2616 and the data plane VCN 2618 may be included in a service tenancy 2619, which may be owned and / or operated by the IaaS provider.

[0349] The control plane VCN 2616 can include a control plane demilitarized zone (DMZ) tier 2620 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers can limit liability and help mitigate breaches. DMZ tier 2620 can also include one or more load balancer (LB) subnets 2622, a control plane app tier 2624 that can include an app subnet 2626, and a control plane data tier 2628 that can include a database (DB) subnet 2630 (e.g., a front-end DB subnet and / or a back-end DB subnet). LB subnet 2622 included in control plane DMZ layer 2620 is communicatively coupled to app subnet 2626 included in control plane app layer 2624 and to Internet gateway 2634, which may be included in control plane VCN 2616, and app subnet 2626 may be communicatively coupled to DB subnet 2630, service gateway 2636, and network address translation (NAT) gateway 2638, which are included in control plane data layer 2628. Control plane VCN 2616 may include service gateway 2636 and NAT gateway 2638.

[0350] Control plane VCN 2616 can include data plane mirror app layer 2640, which can include app subnet 2626. App subnet 2626 included in data plane mirror app layer 2640 can include virtual network interface controller (VNIC) 2642, which can run compute instance 2644. Compute instance 2644 can communicatively couple app subnet 2626 of data plane mirror app layer 2640 to app subnet 2626, which can be included in data plane app layer 2646.

[0351] Data plane VCN 2618 can include data plane app layer 2646, data plane DMZ layer 2648, and data plane data layer 2650. Data plane DMZ layer 2648 can include LB subnet 2622, which can be communicatively coupled to app subnet 2626 of data plane app layer 2646 and internet gateway 2634 of data plane VCN 2618. App subnet 2626 can be communicatively coupled to service gateway 2636 of data plane VCN 2618 and NAT gateway 2638 of data plane VCN 2618. Data plane data layer 2650 can also include DB subnet 2630, which can be communicatively coupled to app subnet 2626 of data plane app layer 2646.

[0352] The internet gateways 2634 of the control plane VCNs 2616 and data plane VCNs 2618 may be communicatively coupled to a metadata management service 2652, which may be communicatively coupled to the public internet 2654. The public internet 2654 may be communicatively coupled to a NAT gateway 2638 of the control plane VCNs 2616 and data plane VCNs 2618. The service gateways 2636 of the control plane VCNs 2616 and data plane VCNs 2618 may be communicatively coupled to cloud services 2656.

[0353] In some examples, the service gateways 2636 of the control plane VCN 2616 and the data plane VCN 2618 can make application programming interface (API) calls to the cloud services 2656 without going over the public internet 2654. API calls from the service gateway 2636 to the cloud services 2656 can be unidirectional. The service gateway 2636 can make API calls to the cloud services 2656, and the cloud services 2656 can send the requested data to the service gateway 2636. However, the cloud services 2656 do not have to initiate the API calls to the service gateway 2636.

[0354] In some examples, secure host tenancy 2604 may be directly connected to an otherwise isolated service tenancy 2619. Secure host subnet 2608 may communicate with SSH subnet 2614 through LPG 2610, which may allow bidirectional communication through an otherwise isolated system. Connecting secure host subnet 2608 to SSH subnet 2614 allows secure host subnet 2608 to be accessible to other entities within service tenancy 2619.

[0355] The control plane VCN 2616 may enable users of the service tenancy 2619 to configure or provision desired resources. The desired resources provisioned in the control plane VCN 2616 may be deployed or used in the data plane VCN 2618. In some examples, the control plane VCN 2616 may be isolated from the data plane VCN 2618, and the data plane mirror app layer 2640 of the control plane VCN 2616 may communicate with the data plane app layer 2646 of the data plane VCN 2618 via a VNIC 2642 that may be included in the data plane mirror app layer 2640 and the data plane app layer 2646.

[0356] In some examples, a user or customer of the system may make a request (e.g., perform a create, read, update, or delete (CRUD) operation) over the public internet 2654, which may route the request to the metadata management service 2652. The metadata management service 2652 may route the request to the control plane VCN 2616 through the internet gateway 2634. The request may be received by the LB subnet 2622 included in the control plane DMZ tier 2620. The LB subnet 2622 may determine that the request is valid, and in response to this determination, the LB subnet 2622 may send the request to the app subnet 2626 included in the control plane app tier 2624. If the request is validated and a call to the public internet 2654 is required, the call to the public internet 2654 may be sent to the NAT gateway 2638, which may make the call to the public internet 2654. Metadata that may be desirable to store with the request may be stored in the DB subnet 2630.

[0357] In some examples, the data plane mirror app layer 2640 may facilitate direct communication between the control plane VCN 2616 and the data plane VCN 2618. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 2618. The control plane VCN 2616 may perform configuration changes, updates, or other suitable modifications to resources by communicating directly with the resources included in the data plane VCN 2618 via the VNIC 2642.

[0358] In some embodiments, the control plane VCN 2616 and the data plane VCN 2618 may be included in the service tenancy 2619. In this case, a user or customer of the system may not own or operate either the control plane VCN 2616 or the data plane VCN 2618. Alternatively, an IaaS provider may own or operate both the control plane VCN 2616 and the data plane VCN 2618, and both may be included in the service tenancy 2619. This embodiment may enable network isolation that may prevent users or customers from interacting with the resources of other users or customers. This embodiment may also enable private storage of databases by users or customers of the system without having to rely on the public internet 2654, which may not have the desired level of threat protection for storage.

[0359] In another embodiment, LB subnet 2622 included in control plane VCN 2616 can be configured to receive signals from service gateway 2636. In this embodiment, control plane VCN 2616 and data plane VCN 2618 can be configured to be called by customers of the IaaS provider without calling the public internet 2654. Customers of the IaaS provider may desire this embodiment because the databases they use can be stored in service tenancy 2619, which is controlled by the IaaS provider and can be isolated from the public internet 2654.

[0360] 27 is a block diagram 2700 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2702 (e.g., service operator 2602 of FIG. 26) may be communicatively coupled to a virtual cloud network (VCN) 2706 (e.g., VCN 2606 of FIG. 26) and a secure host tenancy 2704 (e.g., secure host tenancy 2604 of FIG. 26), which may include a virtual cloud network (VCN) 2706 (e.g., VCN 2606 of FIG. 26) and a secure host subnet 2708 (e.g., secure host subnet 2608 of FIG. 26). VCN 2706 may include a local peering gateway (LPG) 2710 (e.g., LPG 2610 of FIG. 26), which may be communicatively coupled to a secure shell (SSH) VCN 2712 via an LPG 2610 included in the SSH VCN 2712 (e.g., SSH VCN 2612 of FIG. 26). SSH VCN 2712 can include SSH subnet 2714 (e.g., SSH subnet 2614 in FIG. 26 ), and SSH VCN 2712 can be communicatively coupled to control plane VCN 2716 via LPG 2710, which is included in control plane VCN 2716 (e.g., control plane VCN 2616 in FIG. 26 ). Control plane VCN 2716 can be included in service tenancy 2719 (e.g., service tenancy 2619 in FIG. 26 ), and data plane VCN 2718 (e.g., data plane VCN 2618 in FIG. 26 ) can be included in customer tenancy 2721, which can be owned or operated by a user or customer of the system.

[0361] The control plane VCN 2716 may include a control plane DMZ layer 2720 (e.g., control plane DMZ layer 2620 of FIG. 26 ) that may include a LB subnet 2722 (e.g., LB subnet 2622 of FIG. 26 ), a control plane app layer 2724 (e.g., control plane app layer 2624 of FIG. 26 ) that may include an app subnet 2726 (e.g., app subnet 2626 of FIG. 26 ), and a control plane data layer 2728 (e.g., control plane data layer 2628 of FIG. 26 ) that may include a database (DB) subnet 2730 (e.g., similar to database subnet 2630 of FIG. 26 ). LB subnet 2722 included in control plane DMZ layer 2720 is communicatively coupled to app subnet 2726 included in control plane app layer 2724 and to Internet gateway 2734 (e.g., Internet gateway 2634 in FIG. 26 ), which may be included in control plane VCN 2716, and app subnet 2726 may be communicatively coupled to DB subnet 2730, service gateway 2736 (e.g., service gateway 2636 in FIG. 26 ), and network address translation (NAT) gateway 2738 (e.g., NAT gateway 2638 in FIG. 26 ), which are included in control plane data layer 2728. Control plane VCN 2716 may comprise service gateway 2736 and NAT gateway 2738.

[0362] Control plane VCN 2716 may include a data plane mirror app layer 2740 (e.g., data plane mirror app layer 2640 of FIG. 26 ), which may include an app subnet 2726. App subnet 2726 included in data plane mirror app layer 2740 may include a virtual network interface controller (VNIC) 2742 (e.g., VNIC 2642) on which compute instance 2744 (e.g., similar to compute instance 2644 of FIG. 26 ) may run. Compute instance 2744 may facilitate communication between app subnet 2726 of data plane mirror app layer 2740 and app subnet 2726, which may be included in data plane app layer 2746, via VNIC 2742 included in data plane mirror app layer 2740 and VNIC 2742 included in data plane app layer 2746 (e.g., data plane app layer 2646 of FIG. 26 ).

[0363] An internet gateway 2734 included in the control plane VCN 2716 may be communicatively coupled to a metadata management service 2752 (e.g., metadata management service 2652 of FIG. 26), which may be communicatively coupled to the public internet 2754 (e.g., public internet 2654 of FIG. 26). The public internet 2754 may be communicatively coupled to a NAT gateway 2738 included in the control plane VCN 2716. A service gateway 2736 included in the control plane VCN 2716 may be communicatively coupled to cloud services 2756 (e.g., cloud services 2656 of FIG. 26).

[0364] In some examples, data plane VCN 2718 may be included in customer tenancy 2721. In this case, the IaaS provider may provide a control plane VCN 2716 for each customer, and the IaaS provider may configure a unique compute instance 2744 for each customer, which may be included in service tenancy 2719. Each compute instance 2744 may enable communication between the control plane VCN 2716 in service tenancy 2719 and the data plane VCN 2718 in customer tenancy 2721. The compute instance 2744 may enable deployment or use of resources provisioned in the control plane VCN 2716 in service tenancy 2719 in the data plane VCN 2718 in customer tenancy 2721.

[0365] In another example, a customer of the IaaS provider may have a database that resides in customer tenancy 2721. In this example, control plane VCN 2716 may include data plane mirror app tier 2740, which may include app subnet 2726. Data plane mirror app tier 2740 may reside in data plane VCN 2718, but may not reside in data plane VCN 2718. That is, data plane mirror app tier 2740 may be accessible to customer tenancy 2721, but may not reside in data plane VCN 2718, and may not be owned or operated by the IaaS provider's customer. Data plane mirror app tier 2740 may be configured to make calls to data plane VCN 2718, but may not be configured to make calls to any entities included in control plane VCN 2716. A customer may desire deployment or use of resources in data plane VCN 2718 that have been provisioned in control plane VCN 2716, and data plane mirror app layer 2740 may facilitate the deployment or other use of the resources desired by the customer.

[0366] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 2718. In this embodiment, the customer can determine what the data plane VCN 2718 can access, and the customer may choose to restrict access from the data plane VCN 2718 to the public internet 2754. The IaaS provider may not be able to apply filters or control the data plane VCN 2718's access to any external networks or databases. The customer's application of filters and controls to the data plane VCN 2718 included in customer tenancy 2721 can help isolate the data plane VCN 2718 from other customers and the public internet 2754.

[0367] In some embodiments, calls made by service gateway 2736 allow cloud services 2756 to access services that may not reside on the public internet 2754, the control plane VCN 2716, or the data plane VCN 2718. The connection between cloud services 2756 and the control plane VCN 2716 or the data plane VCN 2718 may not be live or continuous. Cloud services 2756 may reside on different networks owned or operated by the IaaS provider. Cloud services 2756 may be configured to receive calls from service gateway 2736 or may not be configured to receive calls from the public internet 2754. Some cloud services 2756 may be isolated from other cloud services 2756, and control plane VCNs 2716 may be isolated from cloud services 2756 that may not be in the same region as the control plane VCN 2716. For example, control plane VCN 2716 may be located in "Region 1," and cloud service "deployment 26" may be located in Region 1 and Region 2. If a call to deployment 26 is made by service gateway 2736 included in control plane VCN 2716 located in Region 1, the call may be sent to deployment 26 in Region 1. In this example, control plane VCN 2716 or deployment 26 in Region 1 may not be communicatively coupled to or in communication with deployment 26 in Region 2.

[0368] 28 is a block diagram 2800 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2802 (e.g., service operator 2602 of FIG. 26) may be communicatively coupled to a secure host tenancy 2804 (e.g., secure host tenancy 2604 of FIG. 26), which may include a virtual cloud network (VCN) 2806 (e.g., VCN 2606 of FIG. 26) and a secure host subnet 2808 (e.g., secure host subnet 2608 of FIG. 26). VCN 2806 may comprise an LPG 2810, which may be communicatively coupled to an SSH VCN 2812 via an LPG 2810 (e.g., LPG 2610 of FIG. 26) included in SSH VCN 2812 (e.g., SSH VCN 2612 of FIG. 26). SSH VCN 2812 may include SSH subnet 2814 (e.g., SSH subnet 2614 in FIG. 26 ), and SSH VCN 2812 may be communicatively coupled to control plane VCN 2816 via LPG 2810 included in control plane VCN 2816 (e.g., control plane VCN 2616 in FIG. 26 ), and may be communicatively coupled to data plane VCN 2818 via LPG 2810 included in data plane VCN 2818 (e.g., data plane VCN 2618 in FIG. 26 ). Control plane VCN 2816 and data plane VCN 2818 may be included in service tenancy 2819 (e.g., service tenancy 2619 in FIG. 26 ).

[0369] The control plane VCN 2816 may include a control plane DMZ layer 2820 (e.g., control plane DMZ layer 2620 of FIG. 26 ) that may include a load balancer (LB) subnet 2822 (e.g., LB subnet 2622 of FIG. 26 ), a control plane app layer 2824 (e.g., control plane app layer 2624 of FIG. 26 ) that may include an app subnet 2826 (e.g., similar to app subnet 2626 of FIG. 26 ), and a control plane data layer 2828 (e.g., control plane data layer 2628 of FIG. 26 ) that may include a DB subnet 2830. LB subnet 2822 included in control plane DMZ layer 2820 is communicatively coupled to app subnet 2826 included in control plane app layer 2824 and to Internet gateway 2834 (e.g., Internet gateway 2634 in FIG. 26 ), which may be included in control plane VCN 2816, and app subnet 2826 may be communicatively coupled to DB subnet 2830, service gateway 2836 (e.g., service gateway in FIG. 26 ), and network address translation (NAT) gateway 2838 (e.g., NAT gateway 2638 in FIG. 26 ), which are included in control plane data layer 2828. Control plane VCN 2816 may comprise service gateway 2836 and NAT gateway 2838.

[0370] Data plane VCN 2818 may include a data plane app layer 2846 (e.g., data plane app layer 2646 in FIG. 26 ), a data plane DMZ layer 2848 (e.g., data plane DMZ layer 2648 in FIG. 26 ), and a data plane data layer 2850 (e.g., data plane data layer 2650 in FIG. 26 ). Data plane DMZ layer 2848 may include a LB subnet 2822 that may be communicatively coupled to trusted app subnet 2860 and untrusted app subnet 2862 of data plane app layer 2846 and to an internet gateway 2834 included in data plane VCN 2818. Trusted app subnet 2860 may be communicatively coupled to a service gateway 2836 included in data plane VCN 2818, a NAT gateway 2838 included in data plane VCN 2818, and a DB subnet 2830 included in data plane data layer 2850. Untrusted app subnet 2862 may be communicatively coupled to a service gateway 2836 included in data plane VCN 2818 and a DB subnet 2830 included in data plane data layer 2850. Data plane data layer 2850 may include DB subnet 2830, which may be communicatively coupled to a service gateway 2836 included in data plane VCN 2818.

[0371] Untrusted app subnet 2862 may include one or more primary VNICs 2864(1)-2864(N), which may be communicatively coupled to tenant virtual machines (VMs) 2866(1)-2866(N). Each tenant VM 2866(1)-2866(N) may be communicatively coupled to a respective app subnet 2867(1)-2867(N), which may be included in a respective container egress VCN 2868(1)-2868(N), which may be included in a respective customer tenancy 2870(1)-2870(N). Secondary VNICs 2872(1)-2872(N), respectively, may facilitate communication between untrusted app subnet 2862, which is included in data plane VCN 2818, and the app subnets included in the respective container egress VCNs 2868(1)-2868(N). Each container output VCN 2868(1)-2868(N) may include a NAT gateway 2838 that may be communicatively coupled to the public internet 2854 (e.g., public internet 2654 of FIG. 26).

[0372] An internet gateway 2834 included in the control plane VCN 2816 and the data plane VCN 2818 may be communicatively coupled to a metadata management service 2852 (e.g., metadata management system 2652 of FIG. 26 ), which may be communicatively coupled to the public internet 2854. The public internet 2854 may be communicatively coupled to a NAT gateway 2838 included in the control plane VCN 2816 and the data plane VCN 2818. A service gateway 2836 included in the control plane VCN 2816 and the data plane VCN 2818 may be communicatively coupled to cloud services 2856.

[0373] In some embodiments, data plane VCN 2818 may be integrated with customer tenancy 2870. This integration may be useful or desirable for customers of an IaaS provider, such as when they may want support for running code. A customer may provide code to run, which may be destructive, may communicate with other customer resources, or may have undesirable effects. In response, the IaaS provider may determine whether to run the code provided to it by the customer.

[0374] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider to request functionality provided in data plane app layer 2846. The code that performs this functionality may be configured to run in VMs 2866(1)-2866(N) and may not be configured to run elsewhere on data plane VCN 2818. Each VM 2866(1)-2866(N) may be connected to one customer tenancy 2870. Each container 2871(1)-2871(N) contained in VMs 2866(1)-2866(N) may be configured to run code. In this case, double isolation may exist (e.g., containers 2871(1)-2871(N) executing code may be contained in VMs 2866(1)-2866(N) included in at least untrusted app subnet 2862), which may help prevent erroneous or unwanted code from damaging the IaaS provider's network or a different customer's network. Containers 2871(1)-2871(N) may be communicatively coupled to customer tenancy 2870 and may be configured to send or receive data from customer tenancy 2870. Containers 2871(1)-2871(N) may be configured not to send or receive data from any other entity in data plane VCN 2818. Upon completion of code execution, the IaaS provider may disable or discard containers 2871(1)-2871(N).

[0375] In some embodiments, trusted app subnet 2860 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 2860 may be communicatively coupled to DB subnet 2830 and may be configured to perform CRUD operations on DB subnet 2830. Untrusted app subnet 2862 may be communicatively coupled to DB subnet 2830, but in this embodiment, may be configured to perform read operations on DB subnet 2830. Containers 2871(1)-2871(N) included in each customer's VMs 2866(1)-2866(N) and that may execute code from the customer may not be communicatively coupled to DB subnet 2830.

[0376] In other embodiments, the control plane VCN 2816 and the data plane VCN 2818 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 2816 and the data plane VCN 2818. However, communication may occur indirectly in at least one manner. An IaaS provider may establish an LPG 2810 that may facilitate communication between the control plane VCN 2816 and the data plane VCN 2818. In another example, the control plane VCN 2816 or the data plane VCN 2818 may make a call to a cloud service 2856 via a service gateway 2836. For example, a call from the control plane VCN 2816 to the cloud service 2856 may include a request for a service that may communicate with the data plane VCN 2818.

[0377] 29 is a block diagram 2900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2902 (e.g., service operator 2602 of FIG. 26) may be communicatively coupled to a secure host tenancy 2904 (e.g., secure host tenancy 2604 of FIG. 26), which may include a virtual cloud network (VCN) 2906 (e.g., VCN 2606 of FIG. 26) and a secure host subnet 2908 (e.g., secure host subnet 2608 of FIG. 26). VCN 2906 may comprise an LPG 2910, which may be communicatively coupled to an SSH VCN 2912 via an LPG 2910 (e.g., LPG 2610 of FIG. 26) included in SSH VCN 2912 (e.g., SSH VCN 2612 of FIG. 26). SSH VCN 2912 may include SSH subnet 2914 (e.g., SSH subnet 2614 in FIG. 26 ), and SSH VCN 2912 may be communicatively coupled to control plane VCN 2916 via LPG 2910 included in control plane VCN 2916 (e.g., control plane VCN 2616 in FIG. 26 ), and may be communicatively coupled to data plane VCN 2918 via LPG 2910 included in data plane VCN 2918 (e.g., data plane VCN 2618 in FIG. 26 ). Control plane VCN 2916 and data plane VCN 2918 may be included in service tenancy 2919 (e.g., service tenancy 2619 in FIG. 26 ).

[0378] The control plane VCN 2916 may include a control plane DMZ layer 2920 (e.g., control plane DMZ layer 2620 of FIG. 26 ) that may include a LB subnet 2922 (e.g., LB subnet 2622 of FIG. 26 ), a control plane app layer 2924 (e.g., control plane app layer 2624 of FIG. 26 ) that may include an app subnet 2926 (e.g., app subnet 2626 of FIG. 26 ), and a control plane data layer 2928 (e.g., control plane data layer 2628 of FIG. 26 ) that may include a DB subnet 2930 (e.g., DB subnet 2830 of FIG. 28 ). LB subnet 2922 included in control plane DMZ layer 2920 is communicatively coupled to app subnet 2926 included in control plane app layer 2924 and to Internet gateway 2934 (e.g., Internet gateway 2634 in FIG. 26 ), which may be included in control plane VCN 2916, and app subnet 2926 may be communicatively coupled to DB subnet 2930, service gateway 2936 (e.g., service gateway in FIG. 26 ), and network address translation (NAT) gateway 2938 (e.g., NAT gateway 2638 in FIG. 26 ), which are included in control plane data layer 2928. Control plane VCN 2916 may comprise service gateway 2936 and NAT gateway 2938.

[0379] Data plane VCN 2918 may include a data plane app layer 2946 (e.g., data plane app layer 2646 in FIG. 26 ), a data plane DMZ layer 2948 (e.g., data plane DMZ layer 2648 in FIG. 26 ), and a data plane data layer 2950 (e.g., data plane data layer 2650 in FIG. 26 ). Data plane DMZ layer 2948 may include a trusted app subnet 2960 (e.g., trusted app subnet 2860 in FIG. 28 ) and a non-trusted app subnet 2962 (e.g., non-trusted app subnet 2862 in FIG. 28 ) of data plane app layer 2946 and an LB subnet 2922 that may be communicatively coupled to an Internet gateway 2934 included in data plane VCN 2918. Trusted app subnet 2960 may be communicatively coupled to service gateway 2936 included in data plane VCN 2918, NAT gateway 2938 included in data plane VCN 2918, and DB subnet 2930 included in data plane data layer 2950. Untrusted app subnet 2962 may be communicatively coupled to service gateway 2936 included in data plane VCN 2918 and DB subnet 2930 included in data plane data layer 2950. Data plane data layer 2950 may comprise DB subnet 2930, which may be communicatively coupled to service gateway 2936 included in data plane VCN 2918.

[0380] Untrusted app subnet 2962 may include primary VNICs 2964(1)-2964(N) that may be communicatively coupled to tenant virtual machines (VMs) 2966(1)-2966(N) residing within untrusted app subnet 2962. Each tenant VM 2966(1)-2966(N) may execute code in a respective container 2967(1)-2967(N) and may be communicatively coupled to app subnet 2926 that may be included in data plane app layer 2946 that may be included in container egress VCN 2968. Secondary VNICs 2972(1)-2972(N), respectively, may facilitate communication between untrusted app subnet 2962 included in data plane VCN 2918 and the app subnet included in container egress VCN 2968. The container egress VCN may include a NAT gateway 2938 that may be communicatively coupled to the public internet 2954 (e.g., public internet 2654 in FIG. 26).

[0381] An internet gateway 2934 included in the control plane VCN 2916 and the data plane VCN 2918 may be communicatively coupled to a metadata management service 2952 (e.g., metadata management system 2652 of FIG. 26 ), which may be communicatively coupled to the public internet 2954. The public internet 2954 may be communicatively coupled to a NAT gateway 2938 included in the control plane VCN 2916 and the data plane VCN 2918. A service gateway 2936 included in the control plane VCN 2916 and the data plane VCN 2918 may be communicatively coupled to cloud services 2956.

[0382] In some examples, the pattern illustrated by the architecture of block diagram 2900 in FIG. 29 may be considered an exception to the pattern illustrated by the architecture of block diagram 2800 in FIG. 28 and may be desirable for customers of an IaaS provider when the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). Containers 2967(1)-2967(N) contained in VMs 2966(1)-2966(N) for each customer may be accessible in real time by the customer. Containers 2967(1)-2967(N) may be configured to make calls to secondary VNICs 2972(1)-2972(N), respectively, contained in app subnet 2926 of data plane app tier 2946, which may be included in container egress VCN 2968. Secondary VNICs 2972(1)-2972(N) can send the calls to NAT gateway 2938, which may send the calls to public Internet 2954. In this example, containers 2967(1)-2967(N) that are accessible in real time by customers may be isolated from control plane VCN 2916 and may be isolated from other entities included in data plane VCN 2918. Containers 2967(1)-2967(N) may also be isolated from resources of other customers.

[0383] In another example, a customer can use containers 2967(1)-2967(N) to invoke cloud service 2956. In this example, the customer can execute code in containers 2967(1)-2967(N) that requests a service from cloud service 2956. Containers 2967(1)-2967(N) can send the request to secondary VNICs 2972(1)-2972(N), which can send the request to a NAT gateway, which can send the request to public internet 2954. Public internet 2954 can send the request to LB subnet 2922, which is included in control plane VCN 2916, via internet gateway 2934. In response to determining that the request is valid, the LB subnet can send the request to the app subnet 2926, which can send the request to the cloud service 2956 via the service gateway 2936.

[0384] It should be understood that the IaaS architectures 2600, 2700, 2800, and 2900 depicted in the figures may include components other than those shown. Furthermore, the illustrated embodiment is merely an example of a cloud infrastructure system that may incorporate an embodiment of the present disclosure. In other embodiments, an IaaS system may include more or fewer components than those depicted, may combine two or more components, or may have a different configuration or arrangement of components.

[0385] In some embodiments, the IaaS system described herein may include a suite of application, middleware, and database services delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. Oracle Cloud Infrastructure (OCI), offered by the present assignee, is an example of such an IaaS system.

[0386] 30 illustrates an exemplary computer system 3000 upon which various embodiments may be implemented. System 3000 may be adapted to implement any of the computer systems described above. As shown in the figure, computer system 3000 includes a processing unit 3004 that communicates with a number of peripheral subsystems via a bus subsystem 3002. The peripheral subsystems may include a processing acceleration unit 3006, an I / O subsystem 3008, a storage subsystem 3018, and a communications subsystem 3024. Storage subsystem 3018 includes a tangible computer-readable storage medium 3022 and a system memory 3010.

[0387] Bus subsystem 3002 provides a mechanism for allowing the various components and subsystems of computer system 3000 to communicate with each other as desired. While bus subsystem 3002 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 3002 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0388] Processing unit 3004, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of computer system 3000. Processing unit 3004 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, processing unit 3004 may be implemented as one or more independent processing units 3032 and / or 3034, each including a single-core or multi-core processor. In other embodiments, processing unit 3004 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0389] In various embodiments, processing unit 3004 executes various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code being executed may reside in processor 3004 and / or in storage subsystem 3018. With suitable programming, processor 3004 can provide the various functions described above. Computer system 3000 may also include a processing acceleration unit 3006, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0390] The I / O subsystem 3008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include a motion-sensing and / or gesture-recognition device such as a Microsoft Kinect® motion sensor that enables a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and voice commands. User interface input devices may also include an eye-gesture recognition device such as a Google Glass® blink detector that detects a user's eye activity (e.g., "blinking" during photography and / or menu selection) and translates the eye gesture as input to an input device (e.g., Google Glass®). The user interface input devices may also include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0391] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0392] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), a liquid crystal display (LCD), or a plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 3000 to a user or to another computer. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, sound output devices, and modems.

[0393] Computer system 3000 may also include a storage subsystem 3018 that provides a tangible, transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 3004, provide the functionality described above. Storage subsystem 3018 may also provide a repository for storing data used in accordance with the present disclosure.

[0394] As shown in the example of Figure 30, storage subsystem 3018 may include various components including a system memory 3010, a computer-readable storage medium 3022, and a computer-readable storage medium reader 3020. The system memory 3010 may store program instructions that may be loaded and executed by the processing unit 3004. The system memory 3010 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. A variety of different types of programs may be loaded into the system memory 3010, including, but not limited to, client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0395] System memory 3010 may also store operating system 3016. Examples of operating system 3016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In some embodiments in which computer system 3000 runs one or more virtual machines, the virtual machines, along with respective guest operating systems (GOS), may be loaded into system memory 3010 and executed by one or more processors or cores of processing unit 3004.

[0396] The system memory 3010 may have a variety of configurations depending on the type of computer system 3000. For example, the system memory 3010 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations may also be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), and others. In some embodiments, the system memory 3010 may include a basic input / output system (BIOS), which contains the basic routines that help to transfer information between elements within the computer system 3000, such as during start-up.

[0397] Computer-readable storage medium 3022 may represent remote, local, fixed, and / or removable storage devices, as well as storage media for temporarily and / or permanently containing and storing computer-readable information used by computer system 3000 (including instructions executable by processing unit 3004 of computer system 3000).

[0398] The computer-readable storage medium 3022 can include any suitable medium (including storage media and communication media) known or used in the art, including, but not limited to, volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information, which may include tangible computer-readable storage media such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory, and other memory technologies, optical storage such as CD-ROM, digital versatile disk (DVD), magnetic storage devices such as magnetic cassettes, magnetic tape, magnetic disk storage, or other tangible computer-readable media.

[0399] By way of example, the computer-readable storage medium 3022 may include a hard disk drive that reads from and writes to non-removable, nonvolatile magnetic media, a magnetic disk drive that reads from and writes to removable, nonvolatile magnetic disks, and an optical disk drive that reads from and writes to removable, nonvolatile optical disks such as CD-ROMs, DVDs, Blu-Ray® disks, or other optical media. The computer-readable storage medium 3022 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, etc. The computer-readable storage medium 3022 may also include solid-state drives (SSDs) based on nonvolatile memory such as flash memory-based SSDs, enterprise flash drives, semiconductor ROMs, and volatile memory-based SSDs such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 3000 .

[0400] Machine-readable instructions executable by one or more processors or cores of processing unit 3004 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage, including volatile memory storage and / or non-volatile storage. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0401] The communications subsystem 3024 provides an interface to other computer systems and networks. The ...

Claims

1. It is a method, To provide one or more CSP-supplied cloud services to one or more customers of a CSP by using a first portion of the infrastructure provided by a Cloud Service Provider (CSP) in the first region, This includes generating a first virtual private label cloud (vPLC) for a first reseller based on the CSP provision infrastructure, wherein generating the first vPLC includes allocating a second portion of the CSP provision infrastructure to the first vPLC. The aforementioned method, By using the first vPLC, one or more first reseller supply cloud services are provided to one or more customers of the first reseller, By using the CSP-provided infrastructure, a first console including a first set of user interfaces (UIs) is provided to one or more of the CSP's customers. The CSP providing infrastructure further includes providing the first reseller with a second console, which includes a second set of UIs, to one or more of the first reseller's customers. The second console is configured in a different manner from the first console.

2. The method according to claim 1, wherein the first console is run by a first server, and the second console is run by a second server different from the first server.

3. The method according to claim 1 or 2, wherein the first console and the second console are run by the same server.

4. The first set of UIs includes a UI customized for the CSP, The method according to claim 1 or 2, wherein the second set of UIs includes a UI customized for the first reseller.

5. The method according to claim 1 or 2, wherein the first console is associated with a first set of endpoints associated with one or more CSP supply cloud services, and endpoints in the first set of endpoints can be invoked via the first console.

6. The method according to claim 1 or 2, wherein the second console is associated with a second set of endpoints associated with the first reseller supply cloud service, and endpoints in the second set of endpoints can be invoked via the second console.

7. The method according to claim 1 or 2, wherein the second console is associated with the namespace of the first vPLC.

8. The method further includes generating a second vPLC for a second reseller based on the CSP provision infrastructure, wherein generating the second vPLC includes allocating a third portion of the CSP provision infrastructure to the second vPLC. The aforementioned method, By using the second vPLC, one or more second reseller supply cloud services are provided to one or more customers of the second reseller, The CSP infrastructure further includes providing the second reseller with a third console, including a third set of UIs, to one or more of the second reseller's customers. The first console, the second console, and the third console are different, according to the method of claim 1 or 2.

9. It is a system, One or more processors, It comprises one or more computer-readable media that store executable instructions for a computer, When the instruction is executed by one or more processors of the computing system, the system will By using the first portion of the Cloud Service Provider (CSP) provided infrastructure in the first region, one or more CSP-supplied cloud services are provided to one or more customers of the CSP. Based on the CSP provision infrastructure, a first virtual private label cloud (vPLC) is generated for the first reseller. Generating the first vPLC includes allocating a second portion of the CSP provision infrastructure to the first vPLC, and the instruction further to the system, By using the first vPLC, one or more first reseller supply cloud services are provided to one or more customers of the first reseller. By using the CSP-provided infrastructure, the CSP can provide one or more of its customers with a first console including a first set of user interfaces (UIs). By using the CSP provision infrastructure, the first reseller is provided with one or more of its customers with a second console including a second set of UIs. The second console is a system different from the first console.

10. The system according to claim 9, wherein the first console is run by a first server, and the second console is run by a second server different from the first server.

11. The system according to claim 9 or 10, wherein the first console and the second console are run by the same server.

12. The first set of UIs includes a UI customized for the CSP, The system according to claim 9 or 10, wherein the second set of UIs includes a UI customized for the first reseller.

13. The first console is associated with a first set of endpoints related to one or more CSP supply cloud services, and endpoints in the first set of endpoints can be invoked via the first console. The system according to claim 9 or 10, wherein the second console is associated with a second set of endpoints associated with the first reseller supply cloud service, and endpoints in the second set of endpoints can be invoked via the second console.

14. The system according to claim 9 or 10, wherein the second console is associated with the namespace of the first vPLC.

15. It is a method, This includes generating a first virtual private label cloud (vPLC) for a first reseller based on a cloud service provider (CSP) infrastructure in a region, and generating the first vPLC includes allocating a first portion of the CSP infrastructure to the first vPLC. The method further includes generating a second vPLC for a second reseller based on the CSP provision infrastructure, wherein generating the second vPLC includes allocating a second portion of the CSP provision infrastructure to the second vPLC. The aforementioned method, By using the first vPLC, one or more first reseller supply cloud services are provided to one or more customers of the first reseller, By using the second vPLC, one or more second reseller supply cloud services are provided to one or more customers of the second reseller, By using the CSP provision infrastructure, the first reseller is provided with one or more of its customers with a first console including a first set of user interfaces (UIs), The CSP providing infrastructure further includes providing the second reseller with a second console, which includes a second set of UIs, to one or more of the second reseller's customers. The second console is configured in a different manner from the first console.

16. The method according to claim 15, wherein the first console is run by a first server, and the second console is run by a second server different from the first server.

17. The method according to claim 15 or 16, wherein the first console and the second console are run by the same server.

18. The first set of UIs includes a UI customized for the first reseller, The method according to claim 15 or 16, wherein the second set of UIs includes a UI customized for the second reseller.

19. The method according to claim 15 or 16, wherein the first console is associated with the namespace of the first vPLC, and the second console is associated with the namespace of the second vPLC.

20. The first console is associated with a first set of endpoints related to the first reseller supply cloud service, and endpoints in the first set of endpoints can be invoked via the first console. The method according to claim 15 or 16, wherein the second console is associated with a second set of endpoints associated with the second reseller supply cloud service, and endpoints in the second set of endpoints can be invoked via the second console.

Images