Data Center Burst Capacity for Hyperscale Workloads

Abstract

In some aspects, the technique may include monitoring a primary load of the data center and a reserve load of the data center. The primary load and the reserve load may be monitored by a computing device. The primary load of the data center may be configured to be powered by one or more primary generator blocks having a primary capacity, and the reserve load of the data center may be configured to be powered by one or more reserve generator blocks having a reserve capacity. The technique may also include detecting when the primary load of the data center exceeds the primary capacity. Additionally, the technique may include connecting the reserve generator block to at least one of the primary generator block and the primary load using a switch of the computing device.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 17 / 903,924, filed September 6, 2022, entitled "BURST DATACENTER CAPACITY FOR HYPERSCALE WORKLOADS," and U.S. Application No. 18 / 074,332, filed December 2, 2022, entitled "BURST DATACENTER CAPACITY FOR HYPERSCALE WORKLOADS," the disclosures of which are incorporated herein by reference in their entireties for all purposes. [Background technology]

[0002] background Power generation can be a limiting factor in building data centers or increasing data center capacity. Data centers are built with redundant power systems to ensure high availability of servers. However, this redundant power capacity is rarely used. Therefore, there is unused power capacity in data centers, and improvements in data center design are desirable. Summary of the Invention

[0003] A brief overview A system of one or more computers can be configured to perform particular operations or actions by having installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform the actions.

[0004] In one general aspect, a computer-implemented method may include monitoring a primary load of a data center and a reserve load of the data center. The monitoring may be performed by a computing device. The primary load of the data center may be configured to be powered by one or more primary generator blocks and one or more reserve generator blocks. The reserve load of the data center may be configured to be powered by the reserve generator blocks. The primary generator blocks may have a primary capacity, and the one or more reserve generator blocks may have a reserve capacity. The method may include detecting, using a computing device, that the primary load of the data center exceeds the primary capacity. The method may include connecting the reserve generator blocks to at least one of the primary generator blocks and the primary load using a switch. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0005] Implementations may include one or more of the following features: The method in which a standby generator block is connected may further include disconnecting a standby load from the standby generator block. The standby load may be disconnected by a circuit breaker controlled by the computing device. The method in which a standby load is disconnected when a combined load exceeds a combined capacity. The combined load may include a primary load and a standby load. The combined capacity may include a primary capacity and a standby capacity. The method in which a standby load is powered by one or more standby generator blocks and a utility connection, wherein the utility connection provides power to at least half of the standby load. The method in which whether a primary load exceeds a primary capacity is detected may include determining that a primary generator block has failed. The method in which the availability of the primary load is 99.999%. The method in which the availability of the standby load is 99.9%. Implementations of the described technology may include hardware, a method or process, or a computer tangible medium.

[0006] In one general aspect, a system may include a non-transitory computer-readable medium having computer-executable program instructions stored thereon. The system may include a processing device communicatively coupled to the non-transitory computer-readable medium for executing the computer-executable program instructions, where executing the computer-executable program instructions configures the processing device to perform operations that may include monitoring a primary load and a reserve load of a data center. The primary load of the data center may be powered by one or more primary generator blocks having a primary capacity. The reserve load of the data center may be configured to be powered by one or more reserve generator blocks having a reserve capacity. The instructions may include detecting that the primary load of the data center exceeds the primary capacity. The instructions may include connecting the reserve generator block to at least one of the primary generator block and the primary load using a switch. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] In one general aspect, a non-transitory computer-readable storage medium having computer-executable program instructions stored thereon may include monitoring a primary load and a reserve load of a data center. The primary load of the data center is powered by one or more primary generator blocks having a primary capacity, and the reserve load of the data center is powered by one or more reserve generator blocks having a reserve capacity. The instructions may include a computing device detecting that the primary load of the data center exceeds the primary capacity. The instructions may include connecting the reserve generator block to at least one of the primary generator block and the primary load using a switch controlled by the computing device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of a block redundant power architecture during normal operation according to an embodiment. [Figure 2] FIG. 1 is a diagram of a block redundant power architecture during a failure scenario according to an embodiment. [Figure 3] FIG. 1 is a diagram of a capacitive harvesting power architecture during normal operation according to an embodiment. [Figure 4] FIG. 1 is a diagram of a capacity harvesting power architecture during a failure scenario according to an embodiment. [Figure 5] FIG. 10 illustrates a method for decoupling a preload from a preblock according to an embodiment. [Figure 6] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 7] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 8] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 9] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an example computer system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description In the following description, various aspects are described. For purposes of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that these embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0010] Data centers are designed to minimize interruptions in server availability. Systems within data centers are often designed with redundancy to ensure that the failure of a single component does not result in an interruption in service. For example, backup generator blocks can supplement the power supplied to data center servers from a connection to the local power grid. In addition, some of the generator blocks are "reserve" or backup generator blocks that may not be connected to a load under normal operating conditions. Thus, if the power grid and the "primary" (i.e., non-backup) generator blocks fail, the reserve generator blocks can still power the servers.

[0011] Redundant data center designs can be wasteful, and redundant systems often have unused capacity sitting idle. For example, a standby generator block may only be activated in the rare event that two other power sources are unavailable. Such a standby generator block may typically be needed to power the primary load for less than nine hours per year. Accepted industry standards are for the load to be available 99.999% of the time, with downtime of less than six minutes per year.

[0012] While some time-sensitive, high-priority loads may require this industry standard, known as five-nines availability, in some situations this standard may be excessive. For example, a server that processes corporate payroll, which may need to be completed in a few days, may not require five-nines availability, as a delay of a few minutes in availability may not be noticed by customers.

[0013] Additionally, five nines (e.g., 99.999%) availability standards are based on assumptions of server downtime using technology that is now outdated. For example, the most recent Institute of Electrical and Electronics Engineers (IEEE) standards for five nines (e.g., IEEE Std. 3006.7, IEEE Recommended Practice for Determining the Reliability of 7x24 Continuous Power Systems in Industrial and Commercial Facilities) were published in 2013. The calculations in the 3006.7 standard are based on the reliability of power system equipment, such as generators and uninterruptible power supplies (UPS), as of 2013. Improvements in technology since the standard was updated could mean that the time a standby generator block is needed could be even shorter than the approximately nine hours per year suggested by some standards.

[0014] Thus, a data center's standby generator block is a costly resource that is rarely utilized. The standby block can be used to provide backup power for a standby load with a lower availability standard, such as three-nines or 99.9% availability. This standby load can be a dual-corded opportunistic load connected to utility power and the standby generator block. If the primary load exceeds the capacity of the primary generator block, the standby load can be dropped, and the standby generator block can provide power to the primary load. For example, if a component within the primary generator block fails, the primary load could exceed the capacity of the primary generator block. The standby load can be disconnected from the standby generator and continue to receive power unless the utility connection fails.

[0015] In one illustrative example, a primary load, with a service level agreement requiring five-nines availability, is connected to five primary generator blocks. If a primary generator block fails, the primary load of the failed block can be transferred via a switch to a standby generator block. Each generator block includes a utility connection, a generator, and a battery, known as an uninterruptible power supply (UPS), which provides power to the load while the generator is running. A circuit breaker connects the standby load to the standby power block, which is also connected to the local power grid via a utility connection.

[0016] Continuing with this example, utility power to the main generator block is interrupted, causing the primary load to experience a power drop. The main generator block attempts to provide backup power to the primary load, but the block's UPS is inoperative. When the power provided by the failed generator block begins to drop, a switch called a static transfer switch (STS) detects the drop in power from that block. In response, the STS disconnects the main generator block and connects the standby generator to the load, causing the standby block's UPS to provide power until the standby generator starts.

[0017] The standby generator block attempts to power both the standby load and the primary load, but the combined load exceeds the capacity of the standby generator block. To ensure that the primary load continues to receive power, a circuit breaker opens, disconnecting the standby load from the standby generator block so that its power can be supplied to the primary load.

[0018] FIG. 1 is a diagram of a block redundant power architecture 100 during normal operation according to one embodiment. This architecture can be part of a block redundant (BR) architecture, where, under normal operating conditions, a main generator block 105 provides power to a load 110 using a utility power connection 115. While switches in FIGS. 1-4 , such as switch 120, are shown in an open position, this is primarily to illustrate that the switch is in a particular position and should not be construed as indicating an open or closed connection. Instead, switch 120, and any other switches shown herein, can be open or closed and can be controlled to change from an open to a closed or closed to an open state. Normal operating conditions can include a utility power interruption, and if the utility power is interrupted, the main generator block can provide power to the load 110 using a generator 125 in addition to an uninterruptible power supply (UPS) 130. The load, such as load 110, can be one or more electronic or computing devices. For example, load 110 can include a server computer, a personal computer, a storage device, a network device, a cooling device, a fan, an environmental monitor, etc. Under normal operating conditions, the load 110 should not exceed the capacity of the generator block 105 .

[0019] The UPS 130 is an electronic device that can provide emergency power when it detects an interruption in utility power. A main generator block may have one or more UPSs 130 and one or more generators 125. The emergency power may be power that can be supplied by the UPS 130 for a short period of time (e.g., 25 milliseconds or less). One or more batteries in the UPS 130 can provide emergency power to the load 110 while the generator 125 is running. The time to readiness, i.e., the time to start the generator 125, may be between 10 and 15 seconds. In addition to providing emergency power, the UPS 130 is a transmission path for power from the utility company. The UPS 130 can also address utility power issues, such as correcting voltage spikes, voltage sags, noise, etc.

[0020] The BR power architecture may include one or more standby (R) blocks 135. The R block 135 may not be active unless the main generator block 105 fails, and the R block 135 may be connected to a static transfer switch (STS) 140 without providing power to the loads 110. The STS 140 may be an electronic device that can transfer power from a main power source to an alternate power source for a short period of time (e.g., 4 milliseconds). The STS 140 may switch to an alternate power source if power from the main power source (e.g., the power source providing power through the STS) falls below a threshold. The loads 110 may be connected to the STS 140 through a switchboard (SWB) 145 that can distribute power to the loads 110. One or more loads may be connected to the main generator block 105, such as an A block 150, a B block 155, or a C block 160, and the SWB 145 can distribute power to the one or more loads. For example, SWB 145 may be used to remove power from a server (e.g., power down the server) in order to perform maintenance on the server. Although three main generator blocks 105 and one R block 135 are shown in architecture 100, other configurations are contemplated, such as configurations with a main generator block to standby block ratio of 3:1, 5:1, 7:1, 12:1, etc.

[0021] FIG. 2 is a diagram 200 of a block redundant power architecture during a failure scenario according to one embodiment. As described above, the main generator blocks 205 provide power to the load 210 under normal operating conditions. A utility power failure may be a normal operating condition, and a failure condition may be when one of the main generator blocks 205 is unable to provide power to the load 210. A failure condition may be when the utility power provided via the utility power connections 215 is interrupted and neither the generators 225 nor the UPS 230 are able to provide emergency power from one or more of the main generator blocks 205. A failure condition may occur when one of the main blocks 205 has a failure in: 1) the utility power connection 215 and the generator 225; 2) the utility power connection 215 and the UPS 230; or 3) the UPS 230. If the utility power connection 215 fails but the UPS 230 and the generator 225 are operational, the UPS 230 can derive its power from the generator 225. If the generator 225 fails but the UPS 230 and utility power connection 214 are operational, the UPS 230 can continue to provide power even without transmission from the utility power connection 215 .

[0022] In the event of a failure of the main generator block 205 (e.g., when a fault condition occurs), the R block 235 may operate to ensure that power is maintained to the load 210. The R block 235 may provide emergency power to the load 210 through the utility power connection 215, or the R block 235 can operate by providing emergency power to the load 210 from the generator 225. If there is a delay in the power provided by the generator 225, the UPS 230 can provide emergency power to the load 210. For example, the generator 225 may be unable to provide power during a start-up process.

[0023] The STS 240 can detect that a primary generator block, such as the A block 250, the B block 255, or the C block 260, is no longer powering the load 210. For example, diagram 200 shows a black "X" indicating a fault in the A block 250. The STS 240 detects a drop in power provided by the A block 250, and the STS 240 switches from its connection with the A block 250 to its connection with the R block 235 (e.g., the dotted connection). The R block can provide power to the load 210 until the faulty block, in this case the A block 250, is returned to service and resumes providing power to the load 210. The R block 235 can provide power to the load 210 during a fault condition, which may include component damage or failure that renders the A block 250 inoperable or scheduled maintenance of the A block 250.

[0024] 3 is a diagram of a capacity harvesting power architecture 300 during normal operation according to one embodiment. Architecture 300 may include a main generator block 305 that supplies power to load 310 and a reserve (R) block 315 that can supply power to a reserve (R) load 320. Although three main blocks 305 and one R block 315 are shown in architecture 300, other configurations are contemplated, such as configurations with main block to reserve block ratios of 3:1, 5:1, 7:1, and 12:1.

[0025] The main generator block 305 and the standby block 315 can supply utility power to the load 310 or the R load 320 via the UPS 325. If utility power to the main generator block is interrupted, the UPS 325 can provide emergency power to the load 310 or the R load 320 during the delay between the activation of the generator 330 and the generator being able to provide enough power to support the load. For example, the turbine in the generator 330 may take several seconds to reach a speed sufficient to generate electricity.

[0026] During normal operation, the main generator block 305, which includes the A block 335, the B block 340, and the C block 345, can supply power to the load 310, and the R block 315 can supply power to the R load 320. The load 310 can be a high-availability load that works in conjunction with the R block 315 to provide 99.999% uptime over a given period of time (e.g., "five nines" availability). The main generator block 305 can provide 99.999% availability because the R block 315 can provide power if one of the main generator blocks 305 fails. A system with 99.999% uptime has a downtime of 5.26 minutes or less per year. Standards for commercial power systems that can provide "five nines" of availability are set forth, for example, in IEEE Std 493-2007 (a revision of IEEE Std 493-1997), "IEEE Recommended Practice for the Design of Reliable Industrial and Commercial Power Systems," Volume N / A, Issue N / A, pp. 1-383, June 25, 2007, doi:10.1109 / IEEESTD.2007.380668.

[0027] The R load 320 may have a lower availability than the load 310. For example, the R load 320 may have an availability of 99.99%, 99.9%, or 99%. Under normal operation, power is supplied to the R load via the utility connection 350 and the utility connection through the R block 315 (e.g., via the UPS 325). 50% of the R load 320 may be supplied by the R block 315, and the remaining 50% may be supplied from the utility connection 350. The R block 315 may provide a greater or lesser percentage of the power delivered to the R load 320 (e.g., 5%, 25%, 45%, 55%, 75%, 95% of the power to the R load). During normal operation, if the R block 315 fails, the utility connection 350 can provide full power to the R load 320. If the utility connection 350 fails, the R block 315 can provide full power to the R load 320.

[0028] FIG. 4 is a diagram of a capacity harvesting power architecture 400 during a failure scenario, according to one embodiment. A failure scenario can occur when one of the main generator blocks 405 fails. A main generator block, such as A block 410, B block 415, or C block 420, can fail when the UPS 425 or generator 430 fails and the Tx 417 fails (although not specified, the Txs may be referred to herein as 412, 417, or 422, in this case, as the failure of the Tx 417). In architecture 400, B block 415 is shown with a black "X" to indicate that the block has failed, but a failure scenario can occur when any of the main blocks fails. A failure scenario can occur when there is a reduction in the power that the main generator block 405 provides to the load 435; for example, a failure scenario can occur when one or more generators and Txs or UPSs in the main generator block fail.

[0029] The R block 440 can be disconnected from one or more R loads 445 when one of the main generator blocks 405 fails and one or more STSs 455 switch from primary to standby (R) power. The R loads may be disconnected when one of the main blocks 405 fails. Primary power can be provided by one of the main generator blocks 405, and standby power can be provided by the standby block (440). The R loads 445 can be disconnected from the R block 440 by a circuit breaker (e.g., breaker 450). If the sum of the load 435 on the main generator block 405 and the R loads 445 exceeds the capacity of the R block 440, the R block 440 can be disconnected from the R loads 445 by the breaker 450. In the event of a failure in one of the main generator blocks 504, the breaker 450 can disconnect one or more of the R loads 445 if the combined loads 435, 445 exceed the capacity of the available blocks (e.g., main generator block 405, R block 440). If the sum of the loads 435 and R loads 445 of a main generator block 405 (e.g., A block 410, B block 415, or C block 420) exceeds the capacity of the R block 445, the R block 440 can be disconnected from the R loads 445 by the breaker 450. The breaker 450 can disconnect the R block 440 from the R loads 445 if a threshold number of STSs 455 switch from primary power to R power (e.g., two or more STSs switch to reserve power).

[0030] The computing device 460 can control the breaker 450 to disconnect or connect the R load 445 to the R block 440. In some embodiments, the computing device 460 can be an industrial control system implemented in hardware rather than software. For example, the industrial control system can be a hardware-implemented control system that disconnects the R block whenever one of the main generator blocks 405 fails. The breaker 450 can be controlled based on signals from the STSs 455; for example, these signals can indicate whether the STSs 455 are connected to primary power or standby (R) power. The computing device 460 can open the breaker 450 if one or more of the STSs 455, or a threshold amount of the STSs, are switching to R power (e.g., switching to receive power from the R block 440). Other techniques for determining whether to open the breaker 450 are also contemplated. For example, the voltage, current, resistance, or inductance between the main generator block 405 and the load 435 may be measured to determine whether the breaker 450 should open (e.g., to determine whether the main generator block 405 is providing enough power to the load 435).

[0031] The computing device 460 may close the breaker 450 if the computing device determines that the R block 440 does not need to supply power to the load 435. For example, the UPS 425 in the B block 415 may have malfunctioned, causing the computing device 460 to open the breaker 450, and after the B block 415 was returned to service, the computing device 460 closed the breaker 450. The computing device 460 may be a programmable logic device, a personal computer, a system on a chip, a single board computer (SBC), a field programmable gate assembly (FPGA), an integrated circuit, a programmable logic circuit (PLC), etc.

[0032] 5 is a diagram of a method for decoupling a spare load from a spare block according to one embodiment. The method is illustrated as a logical flow diagram, each operation of which may be implemented as hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations may represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement a process or method.

[0033] Considering method 500 in more detail, at block 510, primary and secondary loads of a data center may be monitored. The primary loads may be loads 110, 210, 310, 435, and the secondary loads may be R loads 320, 445. The primary and secondary loads may be monitored by a computing device 460. For example, the computing device 460 may receive an output from an STS, such as STS 140, 240, 455, indicating whether the STS is connected to power from a primary block (e.g., 105, 205, 305, 405) or a secondary block, such as secondary block 135, 235, 315, 440. The STS may provide information about the current flowing through the STS to the computing device 460, which may be used to monitor the primary loads. The generator 125, 225, 330, 430, UPS 130, 230, 325, 425, or breaker 450 can provide information to a computing device about the power provided to the primary or secondary load, and the computing device 460 can use the provided information to monitor the primary or secondary load.

[0034] In block 520, it may be determined whether the primary load of the data center exceeds its primary capacity. The primary capacity may be the generating capacity of one or more of the primary generator blocks 105, 205, 305, 405. The primary capacity of the primary block may be the sum of the capacities of the generators 125, 225, 330, 430 in one of the primary generator blocks 102, 205, 305, 405. The reserve capacity of the reserve block may be the sum of the capacities of the generators 125, 225, 330, 430 in the R block 135, 235, 315, 440. The STS 140, 240, 455 may determine that the primary load has exceeded its primary capacity when the power through the STS is below a threshold. The power through the STS may be power provided by one or more of the primary generator blocks.

[0035] In block 530, the standby generator block may be connected to at least one of the primary load or the primary generator block. The primary load may receive power from both the primary generator block and the standby generator block. The standby block may be connected to the primary load when the primary generator block or blocks connected to the primary load are unable to supply the primary load with enough power to keep the primary load operational. The primary load or standby load may include a server computer, a personal computer, a storage device, a network device, a cooling device, a fan, an environmental monitor, etc.

[0036] The STS 140, 240, 455 can connect the standby generator block to the primary load by switching it from the primary generator block to the standby generator block. A generator block, such as the primary generator block 105, 205, 305, 405 or the R-block 135, 235, 315, 440, can be connected to a load by one or more STSs. For example, a generator block can contain multiple generators, and each generator in the block can be connected to a load by an STS. Some or all of the STSs can be switched to transfer a portion of the load from one generator block to another.

[0037] Connecting the standby generator block may include disconnecting the standby load from the standby generator block. The standby load may be disconnected by a circuit breaker, such as breaker 450, that may be controllable by or part of the computing device 460. The utility power connection 115, 215, 350 may supply power to the standby load after the load is disconnected from the standby block. The computing device 460 may close breaker 450 and restore the connection between the standby load and the standby block if the combined load of the primary load and the standby load connected to the standby block does not exceed the combined generating capacity of the primary generator block connected to the primary load and the standby block (e.g., if the standby block has sufficient spare capacity to support the standby load). The standby block may be connected to one or more standby loads by one or more circuit breakers, and the standby block may be reconnected to some or all of the standby loads. The standby block may be reconnected if the primary block, which was inoperable due to component failure or maintenance, is able to resume generating power. The standby loads may have various tiers, with lower tiers being disconnected before higher tiers. Spare loads can be reconnected to power, with higher tiers being reconnected before lower tiers. Tiers can be set based on priority or service level agreements (SLAs), with lower tiers having lower priority and higher tiers having higher priority.

[0038] Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with these infrastructure components. In this case, these services can be policy-driven, allowing IaaS users to implement policies that drive load balancing to maintain application availability and performance.

[0039] In some cases, IaaS customers may access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, etc.

[0040] In most cases, the cloud computing model requires the involvement of a cloud provider, which may, but need not, be a third-party service that specializes in providing (e.g., offering, renting, or selling) IaaS. Alternatively, an entity may choose to deploy a private cloud and become the provider of that infrastructure service.

[0041] In some examples, an IaaS deployment is the process of putting a new application, or a new version of an application, onto a prepared application server, etc. It can also include the process of preparing the server (e.g., installing libraries, daemons, etc.), which is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). In this case, the customer may be responsible for handling the deployment of the (OS), middleware, and / or application (e.g., on self-service virtual machines (e.g., that can be spun up on demand)).

[0042] In some instances, provisioning IaaS may refer to obtaining a computer or virtual host for use and then installing any required libraries or services on it. In most cases, deployment does not include provisioning, which may need to be done first.

[0043] In some cases, IaaS provisioning presents two distinct challenges. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is running. Second, there is the challenge of evolving the existing infrastructure (adding new services, modifying services, removing services, etc.) once everything is provisioned. In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. In this case, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together with each other) can be described declaratively. In some cases, once the topology is defined, workflows can be generated to create and / or manage the various components described in the configuration files.

[0044] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a pool of configurable and / or shared computing resources that may be on-demand), also known as a core network. In some examples, there may also be one or more security group rules and one or more virtual machines (VMs) that are provisioned to define how the security of the network will be configured. Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure may evolve over time as more infrastructure elements are desired and / or added.

[0045] In some cases, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques enable infrastructure management within these environments. In some examples, a service team may write code that they want to deploy to one or more, but often many, different production environments (e.g., across various different geographic locations, sometimes across the globe). However, in some examples, the infrastructure into which the code will be deployed must first be set up. In some cases, provisioning may be done manually, utilizing provisioning tools to provision resources, and / or utilizing deployment tools to deploy the code once the infrastructure has been provisioned.

[0046] 6 is a block diagram 600 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service provider 602 may be communicatively coupled to a secure host tenancy 604, which may include a virtual cloud network (VCN) 606 and a secure host subnet 608. In some examples, the service provider 602 may employ one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), may run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and may support Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, including, by way of example, personal and / or laptop computers running various versions of the Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may also be workstation computers running any of the various commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively, or in addition, the client computing device may be any other electronic device capable of communicating over VCN 606 and / or an Internet-accessible network, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device.

[0047] VCN 606 may include a local peering gateway (LPG) 610 that may be communicatively coupled to a secure shell (SSH) VCN 612 via an LPG 610 included in SSH VCN 612. SSH VCN 612 may include an SSH subnet 614, and SSH VCN 612 may be communicatively coupled to a control plane VCN 616 via an LPG 610 included in control plane VCN 616. SSH VCN 612 may also be communicatively coupled to a data plane VCN 618 via LPG 610. The control plane VCN 616 and the data plane VCN 618 may be included in a service tenancy 619, which may be owned and / or operated by an IaaS provider.

[0048] The control plane VCN 616 may include a control plane demilitarized zone (DMZ) tier 620 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may have limited responsibility and help mitigate security breaches. Additionally, the DMZ tier 620 may include one or more load balancer (LB) subnets 622, a control plane app tier 624 that may include an app subnet 626, and a control plane data tier 628 that may include a database (DB) subnet 630 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 622 included in the control plane DMZ layer 620 can be communicatively coupled to an app subnet 626 included in the control plane app layer 624 and to an Internet gateway 634 that may be included in the control plane VCN 616, and the app subnet 626 can be communicatively coupled to a DB subnet 630 included in the control plane data layer 628, to a service gateway 636, and to a network address translation (NAT) gateway 638. The control plane VCN 616 can include the service gateway 636 and the NAT gateway 638.

[0049] The control plane VCN 616 may include a data plane mirror app layer 640 that may include an app subnet 626. The app subnet 626 included in the data plane mirror app layer 640 may include a virtual network interface controller (VNIC) 642 on which a compute instance 644 can run. The compute instance 644 may communicatively couple the app subnet 626 of the data plane mirror app layer 640 to the app subnet 626 that may be included in the data plane app layer 646.

[0050] The data plane VCN 618 may include a data plane app layer 646, a data plane DMZ layer 648, and a data plane data layer 650. The data plane DMZ layer 648 may include a LB subnet 622, which may be communicatively coupled to an app subnet 626 of the data plane app layer 646 and an Internet gateway 634 of the data plane VCN 618. The app subnet 626 may be communicatively coupled to a service gateway 636 of the data plane VCN 618 and a NAT gateway 638 of the data plane VCN 618. The data plane data layer 650 may also include a DB subnet 630, which may be communicatively coupled to the app subnet 626 of the data plane app layer 646.

[0051] The internet gateways 634 of the control plane VCN 616 and the data plane VCN 618 may be communicatively coupled to a metadata management service 652, which may be communicatively coupled to the public internet 654. The public internet 654 may be communicatively coupled to NAT gateways 638 of the control plane VCN 616 and the data plane VCN 618. The service gateways 636 of the control plane VCN 616 and the data plane VCN 618 may be communicatively coupled to cloud services 656.

[0052] In some examples, a service gateway 636 in the control plane VCN 616 or in the data plane VCN 618 can make application programming interface (API) calls to a cloud service 656 without traversing the public internet 654. API calls from the service gateway 636 to the cloud service 656 can be unidirectional. That is, the service gateway 636 can make an API call to the cloud service 656, and the cloud service 656 can send the requested data to the service gateway 636. However, the cloud service 656 cannot initiate an API call to the service gateway 636.

[0053] In some examples, secure host tenancy 604 can be directly connected to service tenancy 619, which may otherwise be isolated. Secure host subnet 608 can communicate with SSH subnet 614 through LPG 610, which may allow two-way communication on a normally isolated system. Connecting secure host subnet 608 to SSH subnet 614 may allow secure host subnet 608 to access other entities within service tenancy 619.

[0054] The control plane VCN 616 may enable users of the service tenancy 619 to set up or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 616 may be deployed or otherwise used in the data plane VCN 618. In some examples, the control plane VCN 616 may be isolated from the data plane VCN 618, and the data plane mirror app layer 640 of the control plane VCN 616 may communicate with the data plane app layer 646 of the data plane VCN 618 via a VNIC 642, which may be included in the data plane mirror app layer 640 and the data plane app layer 646.

[0055] In some examples, a user or customer of the system may make a request, e.g., a create, read, update, or delete (CRUD) operation, over the public Internet 654, which may communicate the request to a metadata management service 652. The metadata management service 652 may communicate the request to the control plane VCN 616 through an Internet gateway 634. The request may be received by a LB subnet 622 included in the control plane DMZ tier 620. The LB subnet 622 may determine that the request is valid, and in response to this determination, the LB subnet 622 may send the request to an app subnet 626 included in the control plane app tier 624. If the request is validated as valid and requires a call to the public Internet 654, the call to the public Internet 654 may be sent to a NAT gateway 638, which may make the call to the public Internet 654. Memory that may be desired to be saved by the request may be stored in the DB subnet 630.

[0056] In some examples, the data plane mirror app layer 640 may facilitate direct communication between the control plane VCN 616 and the data plane VCN 618. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources included in the data plane VCN 618. Via the VNIC 642, the control plane VCN 616 can communicate directly with resources included in the data plane VCN 618, thereby performing configuration changes, updates, or other appropriate modifications to resources included in the data plane VCN 618.

[0057] In some embodiments, the control plane VCN 616 and the data plane VCN 618 may be included in the service tenancy 619. In this case, a user or customer of the system may not own or operate either the control plane VCN 616 or the data plane VCN 618. Instead, an IaaS provider may own or operate the control plane VCN 616 and the data plane VCN 618, both of which may be included in the service tenancy 619. This embodiment may enable network isolation that may prevent users or customers from interacting with other users' or customers' resources. This embodiment may also enable users or customers of the system to store databases privately without having to rely on the public internet 654, which may not have the desired level of security.

[0058] In another embodiment, the LB subnet 622 included in the control plane VCN 616 may be configured to receive signals from the service gateway 636. In this embodiment, the control plane VCN 616 and the data plane VCN 618 may be configured to be called by the IaaS provider's customers without calling the public internet 654. The IaaS provider's customers may desire this embodiment because databases used by the customers can be controlled by the IaaS provider and stored in the service tenancy 619, which may be isolated from the public internet 654.

[0059] 7 is a block diagram 700 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service provider 702 (e.g., service provider 602 of FIG. 6 ) may be communicatively coupled to a secure host tenancy 704 (e.g., secure host tenancy 604 of FIG. 6 ), which may include a virtual cloud network (VCN) 706 (e.g., VCN 606 of FIG. 6 ) and a secure host subnet 708 (e.g., secure host subnet 608 of FIG. 6 ). VCN 706 may include a local peering gateway (LPG) 710 (e.g., LPG 610 of FIG. 6 ), which may be communicatively coupled to a secure shell (SSH) VCN 712 (e.g., SSH VCN 612 of FIG. 6 ) via an LPG 610 included in the SSH VCN 712. SSH VCN 712 can include an SSH subnet 714 (e.g., SSH subnet 614 in FIG. 6 ), which can be communicatively coupled to a control plane VCN 716 (e.g., control plane VCN 616 in FIG. 6 ) via an LPG 710 included in the control plane VCN 716. The control plane VCN 716 can be included in a service tenancy 719 (e.g., service tenancy 619 in FIG. 6 ), and the data plane VCN 718 (e.g., data plane VCN 618 in FIG. 6 ) can be included in a customer tenancy 721, which can be owned or operated by a user or customer of the system.

[0060] The control plane VCN 716 may include a control plane DMZ layer 720 (e.g., the control plane DMZ layer 620 of FIG. 6 ) that may include a LB subnet 722 (e.g., the LB subnet 622 of FIG. 6 ), a control plane app layer 724 (e.g., the control plane app layer 624 of FIG. 6 ) that may include an app subnet 726 (e.g., the app subnet 626 of FIG. 6 ), and a control plane data layer 728 (e.g., the control plane data layer 628 of FIG. 6 ) that may include a database (DB) subnet 730 (e.g., similar to the DB subnet 630 of FIG. 6 ). The LB subnet 722 included in the control plane DMZ layer 720 can be communicatively coupled to an app subnet 726 included in the control plane app layer 724 and to an Internet gateway 734 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 716, and the app subnet 726 can be communicatively coupled to a DB subnet 730 included in the control plane data layer 728, to a service gateway 736 (e.g., service gateway in FIG. 6 ), and to a network address translation (NAT) gateway 738 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 716 can include the service gateway 736 and the NAT gateway 738.

[0061] The control plane VCN 716 may include a data plane mirror app layer 740 (e.g., data plane mirror app layer 640 of FIG. 6 ), which may include an app subnet 726. The app subnet 726 included in the data plane mirror app layer 740 may include a virtual network interface controller (VNIC) 742 (e.g., VNIC 642) on which a compute instance 744 (e.g., similar to compute instance 644 of FIG. 6 ) can run. The compute instance 744 may facilitate communication between the app subnet 726 of the data plane mirror app layer 740 and the app subnet 726 included in the data plane app layer 746 (e.g., data plane app layer 646 of FIG. 6 ), via the VNIC 742 included in the data plane mirror app layer 740 and the VNIC 742 included in the data plane app layer 746.

[0062] An internet gateway 734 included in the control plane VCN 716 may be communicatively coupled to a metadata management service 752 (e.g., metadata management service 652 in FIG. 6), which may be communicatively coupled to the public internet 754 (e.g., public internet 654 in FIG. 6). The public internet 754 may be communicatively coupled to a NAT gateway 738 included in the control plane VCN 716. A service gateway 736 included in the control plane VCN 716 may be communicatively coupled to cloud services 756 (e.g., cloud services 656 in FIG. 6).

[0063] In some examples, the data plane VCN 718 can be included in the customer tenancy 721. In this case, the IaaS provider may provide a control plane VCN 716 for each customer, and the IaaS provider can configure a unique compute instance 744 for each customer that is included in the service tenancy 719. Each compute instance 744 can enable communication between the control plane VCN 716 included in the service tenancy 719 and the data plane VCN 718 included in the customer tenancy 721. The compute instance 744 can enable resources provisioned in the control plane VCN 716 included in the service tenancy 719 to be deployed or otherwise used in the data plane VCN 718 included in the customer tenancy 721.

[0064] In another example, an IaaS provider customer may have a live database in customer tenancy 721. In this example, control plane VCN 716 may include a data plane mirrored app tier 740, which may include app subnet 726. While data plane mirrored app tier 740 may reside in data plane VCN 718, data plane mirrored app tier 740 may not live in data plane VCN 718. That is, while data plane mirrored app tier 740 may be accessible to customer tenancy 721, data plane mirrored app tier 740 may not reside in data plane VCN 718 or be owned or operated by the IaaS provider customer. Data plane mirrored app tier 740 may be configured to make calls to data plane VCN 718, but may not be configured to make calls to any entities included in control plane VCN 716. A customer may wish to deploy or otherwise use resources in the data plane VCN 718 that have been provisioned in the control plane VCN 716, and the data plane mirror app layer 740 can facilitate the desired deployment or other use of the customer's resources.

[0065] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 718. In this embodiment, the customer can determine what the data plane VCN 718 can access, and the customer can restrict access from the data plane VCN 718 to the public internet 754. The IaaS provider may not be able to filter or otherwise control the data plane VCN 718's access to some external network or database. Applying customer filters and controls to the data plane VCN 718 included in the customer tenancy 721 can help isolate the data plane VCN 718 from other customers and from the public internet 754.

[0066] In some embodiments, cloud services 756 can be invoked by service gateway 736 to access services that may not reside on the public internet 754, the control plane VCN 716, or the data plane VCN 718. The connection between cloud services 756 and the control plane VCN 716 or the data plane VCN 718 may not be constant or continuous. Cloud services 756 may reside on different networks owned or operated by the IaaS provider. Cloud services 756 may be configured to receive calls from service gateway 736 and may not be configured to receive calls from the public internet 754. Some cloud services 756 may be isolated from other cloud services 756, and control plane VCN 716 may be isolated from cloud services 756 that are not in the same region as control plane VCN 716. For example, control plane VCN 716 may be located in “Region 1,” and cloud service “Deployment 6” may be located in “Region 1” and “Region 2.” If a call to deployment 6 is made by a service gateway 736 included in control plane VCN 716 located in region 1, the call may be sent to deployment 6 in region 1. In this example, control plane VCN 716, or deployment 6 in region 1, may not be communicatively coupled to or otherwise in communication with deployment 6 in region 2.

[0067] 8 is a block diagram 800 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service provider 802 (e.g., service provider 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 804 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network (VCN) 806 (e.g., VCN 606 of FIG. 6) and a secure host subnet 808 (e.g., secure host subnet 608 of FIG. 6). VCN 806 may include an LPG 810 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to an SSH VCN 812 (e.g., SSH VCN 612 of FIG. 6) via an LPG 810 included in the SSH VCN 812. SSH VCN 812 can include an SSH subnet 814 (e.g., SSH subnet 614 in FIG. 6 ), and SSH VCN 812 can be communicatively coupled to a control plane VCN 816 (e.g., control plane VCN 616 in FIG. 6 ) via an LPG 810 included in the control plane VCN 816, and to a data plane VCN 818 (e.g., data plane 618 in FIG. 6 ) via an LPG 810 included in the data plane VCN 818. The control plane VCN 816 and the data plane VCN 818 can be included in a service tenancy 819 (e.g., service tenancy 619 in FIG. 6 ).

[0068] The control plane VCN 816 may include a control plane DMZ layer 820 (e.g., the control plane DMZ layer 620 of FIG. 6 ) that may include a load balancer (LB) subnet 822 (e.g., the LB subnet 622 of FIG. 6 ), a control plane app layer 824 (e.g., the control plane app layer 624 of FIG. 6 ) that may include an app subnet 826 (e.g., similar to the app subnet 626 of FIG. 6 ), and a control plane data layer 828 (e.g., the control plane data layer 628 of FIG. 6 ) that may include a DB subnet 830. The LB subnet 822 included in the control plane DMZ layer 820 can be communicatively coupled to an app subnet 826 included in the control plane app layer 824 and to an Internet gateway 834 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 816, and the app subnet 826 can be communicatively coupled to a DB subnet 830 included in the control plane data layer 828 and to a service gateway 836 (e.g., service gateway in FIG. 6 ) and a network address translation (NAT) gateway 838 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 816 can include the service gateway 836 and the NAT gateway 838.

[0069] Data plane VCN 818 may include a data plane app layer 846 (e.g., data plane app layer 646 in FIG. 6 ), a data plane DMZ layer 848 (e.g., data plane DMZ layer 648 in FIG. 6 ), and a data plane data layer 850 (e.g., data plane data layer 650 in FIG. 6 ). Data plane DMZ layer 848 may include LB subnet 822, which may be communicatively coupled to trusted app subnet 860 and untrusted app subnet 862 of data plane app layer 846 and to an Internet gateway 834 included in data plane VCN 818. Trusted app subnet 860 may be communicatively coupled to service gateway 836 included in data plane VCN 818, NAT gateway 838 included in data plane VCN 818, and DB subnet 830 included in data plane data layer 850. Untrusted app subnet 862 may be communicatively coupled to a service gateway 836 included in data plane VCN 818 and a DB subnet 830 included in data plane data layer 850. Data plane data layer 850 may include DB subnet 830, which may be communicatively coupled to a service gateway 836 included in data plane VCN 818.

[0070] The untrusted app subnet 862 may include one or more primary VNICs 864(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 866(1)-(N). Each tenant VM 866(1)-(N) may be communicatively coupled to a corresponding app subnet 867(1)-(N), which may be included in a corresponding container egress VCN 868(1)-(N), which may be included in a corresponding customer tenancy 870(1)-(N). The corresponding secondary VNICs 872(1)-(N) may facilitate communication between the untrusted app subnet 862 included in the data plane VCN 818 and the app subnet included in the container egress VCN 868(1)-(N). Each container egress VCN 868(1)-(N) may include a NAT gateway 838, which may be communicatively coupled to the public internet 854 (e.g., public internet 654 in FIG. 6 ).

[0071] The internet gateway 834 included in the control plane VCN 816 and in the data plane VCN 818 may be communicatively coupled to a metadata management service 852 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public internet 854. The public internet 854 may be communicatively coupled to a NAT gateway 838 included in the control plane VCN 816 and in the data plane VCN 818. The service gateway 836 included in the control plane VCN 816 and in the data plane VCN 818 may be communicatively coupled to cloud services 856.

[0072] In some embodiments, data plane VCN 818 may be integrated with customer tenancy 870. This integration may be useful or desirable in some cases, such as when an IaaS provider's customer may want support for code runtime. Customers may provide code to be executed, but that code may be destructive, may communicate with other customer resources, or may otherwise cause undesirable effects. In response, the IaaS provider can decide whether to run the code that the customer has provided to the IaaS provider.

[0073] In some examples, a customer of an IaaS provider can grant temporary network access to the IaaS provider and request a function to be attached to data plane layer app 846. The code to perform the function may run in VMs 866(1)-(N), which may not be configured to run anywhere else on data plane VCN 818. Each VM 866(1)-(N) may be connected to one customer tenancy 870. Corresponding containers 871(1)-(N) included in VMs 866(1)-(N) may be configured to run the code. In this case, double isolation may exist (e.g., containers 871(1)-(N) may run the code, and containers 871(1)-(N) may be included in at least VMs 866(1)-(N) that are included in untrusted app subnet 862), which may help prevent malformed or otherwise undesirable code from compromising the IaaS provider's network or from compromising another customer's network. Containers 871(1)-(N) can be communicatively coupled to customer tenancy 870 and can be configured to send or receive data from customer tenancy 870. Containers 871(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 818. Once the code execution is complete, the IaaS provider can destroy or otherwise dispose of containers 871(1)-(N).

[0074] In some embodiments, trusted app subnet 860 can execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 860 may be communicatively coupled to DB subnet 830 and configured to perform CRUD operations on DB subnet 830. Untrusted app subnet 862 may be communicatively coupled to DB subnet 830, but in this embodiment, the untrusted app subnet may be configured to perform read operations on DB subnet 830. Containers 871(1)-(N) that may be included in each customer's VMs 866(1)-(N) and that may execute code from the customer may not be communicatively coupled to DB subnet 830.

[0075] In other embodiments, the control plane VCN 816 and the data plane VCN 818 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 816 and the data plane VCN 818. However, communication may occur indirectly through at least one method. The LPG 810 may be established by an IaaS provider that can facilitate communication between the control plane VCN 816 and the data plane VCN 818. In another example, the control plane VCN 816 or the data plane VCN 818 can make a call to a cloud service 856 through the service gateway 836. For example, a call from the control plane VCN 816 to the cloud service 856 may include a request for a service that can communicate with the data plane VCN 818.

[0076] 9 is a block diagram 900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service provider 902 (e.g., service provider 602 of FIG. 6 ) may be communicatively coupled to a secure host tenancy 904 (e.g., secure host tenancy 604 of FIG. 6 ), which may include a virtual cloud network (VCN) 906 (e.g., VCN 606 of FIG. 6 ) and a secure host subnet 908 (e.g., secure host subnet 608 of FIG. 6 ). VCN 906 may include an LPG 910 (e.g., LPG 610 of FIG. 6 ), which may be communicatively coupled to an SSH VCN 912 (e.g., SSH VCN 612 of FIG. 6 ) via an LPG 910 included in the SSH VCN 912. SSH VCN 912 can include an SSH subnet 914 (e.g., SSH subnet 614 in FIG. 6), and SSH VCN 912 can be communicatively coupled to a control plane VCN 916 (e.g., control plane VCN 616 in FIG. 6) via an LPG 910 included in control plane VCN 916, and to a data plane VCN 918 (e.g., data plane 618 in FIG. 6) via an LPG 910 included in data plane VCN 918. Control plane VCN 916 and data plane VCN 918 can be included in a service tenancy 919 (e.g., service tenancy 619 in FIG. 6).

[0077] The control plane VCN 916 may include a control plane DMZ layer 920 (e.g., the control plane DMZ layer 620 of FIG. 6 ) that may include a LB subnet 922 (e.g., the LB subnet 622 of FIG. 6 ), a control plane app layer 924 (e.g., the control plane app layer 624 of FIG. 6 ) that may include an app subnet 926 (e.g., the app subnet 626 of FIG. 6 ), and a control plane data layer 928 (e.g., the control plane data layer 628 of FIG. 6 ) that may include a DB subnet 930 (e.g., the DB subnet 830 of FIG. 8 ). The LB subnet 922 included in the control plane DMZ layer 920 can be communicatively coupled to an app subnet 926 included in the control plane app layer 924 and to an Internet gateway 934 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 916, and the app subnet 926 can be communicatively coupled to a DB subnet 930 included in the control plane data layer 928 and to a service gateway 936 (e.g., service gateway in FIG. 6 ) and a network address translation (NAT) gateway 938 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 916 can include the service gateway 936 and the NAT gateway 938.

[0078] Data plane VCN 918 may include a data plane app layer 946 (e.g., data plane app layer 646 in FIG. 6 ), a data plane DMZ layer 948 (e.g., data plane DMZ layer 648 in FIG. 6 ), and a data plane data layer 950 (e.g., data plane data layer 650 in FIG. 6 ). Data plane DMZ layer 948 may include LB subnet 922, which may be communicatively coupled to a trusted app subnet 960 (e.g., trusted app subnet 860 in FIG. 8 ) and an untrusted app subnet 962 (e.g., untrusted app subnet 862 in FIG. 8 ) of data plane app layer 946 and to an Internet gateway 934 included in data plane VCN 918. Trusted app subnet 960 may be communicatively coupled to a service gateway 936 included in data plane VCN 918, a NAT gateway 938 included in data plane VCN 918, and a DB subnet 930 included in data plane data layer 950. Untrusted app subnet 962 may be communicatively coupled to a service gateway 936 included in data plane VCN 918 and a DB subnet 930 included in data plane data layer 950. Data plane data layer 950 may include DB subnet 930, which may be communicatively coupled to a service gateway 936 included in data plane VCN 918.

[0079] The untrusted app subnet 962 may include primary VNICs 964(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N) residing within the untrusted app subnet 962. Each tenant VM 966(1)-(N) may execute code within a corresponding container 967(1)-(N), which may be communicatively coupled to an app subnet 926, which may be included in a data plane app layer 946, which may be included in a container egress VCN 968. Corresponding secondary VNICs 972(1)-(N) may facilitate communication between the untrusted app subnet 962, which is included in the data plane VCN 918, and the app subnet included in the container egress VCN 968. The container egress VCN may include a NAT gateway 938, which may be communicatively coupled to the public internet 954 (e.g., public internet 654 in FIG. 6 ).

[0080] The internet gateways 934 included in the control plane VCNs 916 and in the data plane VCNs 918 may be communicatively coupled to a metadata management service 952 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public internet 954. The public internet 954 may be communicatively coupled to NAT gateways 938 included in the control plane VCNs 916 and in the data plane VCNs 918. The service gateways 936 included in the control plane VCNs 916 and in the data plane VCNs 918 may be communicatively coupled to cloud services 956.

[0081] In some examples, the pattern illustrated by the architecture of block diagram 900 in FIG. 9 may be considered an exception to the pattern illustrated by the architecture of block diagram 800 in FIG. 8 and may be desirable for an IaaS provider's customers when the IaaS provider cannot communicate directly with the customer (e.g., in isolated regions). Corresponding containers 967(1)-(N) contained in each customer's VMs 966(1)-(N) are accessible to the customer in real time. The containers 967(1)-(N) may be configured to make calls to corresponding secondary VNICs 972(1)-(N) contained in app subnet 926 of data plane app tier 946, which may be contained in container egress VCN 968. The secondary VNICs 972(1)-(N) may send the calls to NAT gateway 938, which can send the calls to public Internet 954. In this example, the containers 967(1)-(N) accessible to the customer in real time may be isolated from the control plane VCN 916 and from other entities contained in the data plane VCN 918. Containers 967(1)-(N) may be isolated from resources from other customers.

[0082] In another example, a customer can invoke cloud service 956 using containers 967(1)-(N). In this example, the customer may execute code in containers 967(1)-(N) that requests a service from cloud service 956. Containers 967(1)-(N) can send the request to secondary VNICs 972(1)-(N), which can send the request to a NAT gateway that can send the request to public internet 954. Public internet 954 can send the request to LB subnet 922, which is included in control plane VCN 916, via internet gateway 934. In response to determining that the request is valid, the LB subnet can send the request to app subnet 926, which can send the request to cloud service 956 via service gateway 936.

[0083] It should be appreciated that the IaaS architectures 600, 700, 800, 900 depicted in the figures may have components other than those depicted. Additionally, the illustrated embodiments are only a few examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In other embodiments, the IaaS systems may have more or fewer components than those depicted in the figures, may combine two or more components, or may have components configured or arranged differently.

[0084] In one embodiment, the IaaS system described herein may include a suite of application, middleware, and database service offerings that are self-service, subscription-based, elastically scalable, and delivered to customers in a reliable, highly available, and secure manner. An example of such an IaaS system is the present assignee's Oracle Cloud Infrastructure (OCI).

[0085] 10 illustrates an exemplary computer system 1000 upon which various embodiments may be implemented. System 1000 may be used to implement any of the computer systems described above. As shown, computer system 1000 includes a processing unit 1004 that communicates with a number of peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. Storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and a system memory 1010.

[0086] Bus subsystem 1002 provides a mechanism for allowing the various components and subsystems of computer system 1000 to communicate with each other as intended. While bus subsystem 1002 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus 1002 may be any of several types of bus structures using any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus conforming to the IEEE P1386.1 standard.

[0087] The processing unit 1004 may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers) and controls the operation of the computer system 1000. The processing unit 1004 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, with each processing unit including a single-core or multi-core processor. In other embodiments, the processing unit 1004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0088] In various embodiments, processing unit 1004 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed may be located in processor 1004 and / or storage subsystem 1018. With appropriate programming, processor 1004 may perform various functionality as described above. Computer system 1000 may additionally include a processing acceleration unit 1006, which may include a digital signal processor (DSP), special purpose processor, and / or the like.

[0089] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, a motion sensing and / or gesture recognition device, such as a Microsoft Kinect® motion sensor, which allows a user to control and interact with an input device, such as a Microsoft Xbox® 360 game controller, through a natural user interface using gestures and voice commands. User interface input devices may also include an eye gesture recognition device, such as a Google Glass® blink detector, which detects a user's eye activity (e.g., "blinking" during photo taking and / or menu selection) and translates the eye gesture as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0090] User interface input devices can also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices can include medical imaging input devices, such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound. User interface input devices can also include audio input devices, such as MIDI keyboards, digital musical instruments, and the like.

[0091] User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, a touch screen, or the like. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. For example, user interface output devices may include various display devices that visually convey text, graphics, and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0092] Computer system 1000 may include a storage subsystem 1018 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data structures that implement the functionality of embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 1004, implement the functionality described above. Storage subsystem 1018 may also provide a repository for storing data used in accordance with the present disclosure.

[0093] 10, the storage subsystem 1018 may include various components including a system memory 1010, a computer-readable storage medium 1022, and a computer-readable storage medium reader 1020. The system memory 1010 may store program instructions loadable and executable by the processing unit 1004. The system memory 1010 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. A variety of different types of programs may be loaded into the system memory 1010, including, but not limited to, client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0094] The system memory 1010 may also store an operating system 1016. Examples of the operating system 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In particular implementations in which the computer system 1000 runs one or more virtual machines, the virtual machines along with their guest operating systems (GOS) may be loaded into the system memory 1010 and executed by one or more processors or cores of the processing unit 1004.

[0095] The system memory 1010 may have different configurations depending on the type of computer system 1000. For example, the system memory 1010 may be volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read-only memory (ROM), flash memory, etc.). Various types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), and others. In some implementations, the system memory 1010 may include a basic input / output system (BIOS), which contains the basic routines that help transfer information between elements within the computer system 1000, such as during start-up.

[0096] Computer-readable storage medium 1022 may represent remote, local, fixed, and / or removable storage devices and storage media for temporarily and / or more permanently storing and storing computer-readable information used by computer system 1000, including instructions executable by processing unit 1004 of computer system 1000.

[0097] The computer-readable storage medium 1022 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and nonvolatile, removable and non-removable media, implemented in any method or technology for information storage and / or transmission. This may include tangible computer-readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage storage, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other tangible computer-readable media.

[0098] By way of example, the computer-readable storage medium 1022 may include hard disk drives that read from or write to non-removable, non-volatile magnetic media, magnetic disk drives that read from or write to removable, non-volatile magnetic disks, and optical disk drives that read from or write to removable, non-volatile optical disks or other optical media, such as CD-ROMs, DVDs, and Blu-Ray® disks. The computer-readable storage medium 1022 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, etc. The computer-readable storage medium 1022 may also include solid-state drives (SSDs) that utilize non-volatile memory, such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, and SSDs that utilize volatile memory, such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 1000.

[0099] Machine-readable instructions executable by one or more processors or cores of the processing unit 1004 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0100] The communications subsystem 1024 provides an interface with other computer systems and networks. The communications subsystem 1024 serves as an interface for receiving data from the computer system 1000 and transmitting data from the computer system 1000 to other systems. For example, the communications subsystem 1024 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1024 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family standards, or other mobile communications technologies, or any combination thereof). In some embodiments, the communications subsystem 1024 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0101] In some embodiments, the communications subsystem 1024 may also receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc., on behalf of one or more users who may be using the computer system 1000.

[0102] By way of example, the communications subsystem 1024 may be configured to receive data feeds 1026 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0103] Additionally, the communications subsystem 1024 may also be configured to receive data in the form of a continuous data stream. This data may include an event stream 1028 of real-time events and / or event updates 1030, which may be continuous or open-ended in nature with no explicit end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc.

[0104] The communications subsystem 1024 may also be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. to one or more databases in communication with one or more streaming data source computers coupled to the computer system 1000.

[0105] The computer system 1000 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0106] Given the ever-changing nature of computers and networks, the description of computer system 1000 depicted in the figures is intended as a specific example only. Many other configurations are possible, having more or fewer components than the system depicted in the figures. For example, customized hardware may be used, and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various embodiments.

[0107] Although specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are encompassed within the scope of the present disclosure. The embodiments are not limited to operation in a particular data processing environment, but can freely operate in multiple data processing environments. In addition, while the embodiments have been described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that the scope of the present disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the above-described embodiments may be used individually or jointly.

[0108] Furthermore, while embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be performed on the same processor or different processors in any combination. Thus, when a component or module is described as being configured to perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or any combination thereof. Processes may communicate using various techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0109] The specification and drawings should therefore be regarded in an illustrative and not a restrictive sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and alterations may be made to the specification and drawings without departing from the broader spirit and scope of the appended claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0110] The terms "a," "an," and "the," and similar referents, in the context of describing the disclosed embodiments, should be construed to encompass both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended (i.e., meaning "including, but not limited to"), unless otherwise noted. The term "connected" should be construed as including, attached to, or connected together in whole or in part, even if there are intervening elements. The recitation of ranges of values ​​herein is intended to serve merely as a shorthand method of individually referring to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The use of language indicating any and all examples or illustrations (e.g., "such as") provided herein is intended merely to better describe the embodiments and does not pose a limitation on the scope of the disclosure unless otherwise asserted. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0111] Disjunctive language such as "at least one of X, Y, or Z," unless expressly stated otherwise, is intended to be understood in context as being used to generally state that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and does not, imply that at least one of X, at least one of Y, or at least one of Z must each be present in an embodiment.

[0112] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art from reading the foregoing description. Such variations can, of course, be accommodated by those skilled in the art, and the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto to the extent permitted by applicable law. Furthermore, unless otherwise indicated herein, this disclosure includes any combination of the above-described elements in all possible variations thereof.

[0113] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0114] While aspects of the disclosure have been described in the foregoing specification with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are therefore to be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method comprising: a computing device monitoring a primary load of a data center and a backup load of said data center; The primary load of the data center is configured to be powered by one or more primary generator blocks having a primary capacity, the primary load being smaller than the primary capacity, and the backup load of the data center is configured to be powered by one or more standby generator blocks having a standby capacity; The method comprises: subsequent to at least one of the increase in the primary load or the decrease in the primary capacity, the computing device detecting that the primary load of the data center exceeds the primary capacity; a switch of the computing device connecting the one or more standby generator blocks to at least one of the one or more main generator blocks or the main load; subsequent to connecting the one or more standby generator blocks to at least one of the one or more main generator blocks or the primary load, the one or more standby generator blocks are connected to (a) the standby load, and (b) at least one of the one or more main generator blocks or the primary load.

2. 2. The method of claim 1, wherein connecting the one or more standby generator blocks further comprises a circuit breaker of the computing device disconnecting the standby load from the one or more standby generator blocks.

3. The method of claim 2 , wherein the auxiliary load is disconnected when a combined load including the primary load and the auxiliary load exceeds a combined capacity including the primary capacity and the auxiliary capacity.

4. 4. The method of claim 1, wherein the reserve loads are supplied with power by the one or more reserve generator blocks and a utility connection, the utility connection providing power to at least half of the reserve loads.

5. The method of any one of claims 1 to 3, wherein detecting that the primary load exceeds the primary capacity further comprises determining that a fault has occurred in one main generator block of the one or more main generator blocks.

6. The method according to any one of claims 1 to 3, wherein the primary load is supplied by the one or more main generator blocks or the one or more standby generator blocks with an availability of 99.999%.

7. The method according to any one of claims 1 to 3, wherein the reserve load is supplied by the one or more reserve generator blocks having an availability of 99.9%.

8. 1. A system comprising: a non-transitory computer-readable medium having computer-executable program instructions stored thereon; a processing device communicatively coupled to the non-transitory computer-readable medium to execute the computer-executable program instructions, wherein executing the computer-executable program instructions configures the processing device to perform operations, the operations including: a computing device monitoring a primary load of a data center and a backup load of the data center, the primary load of the data center being configured to be powered by one or more primary generator blocks having a primary capacity, the primary load being smaller than the primary capacity, and the backup load of the data center being configured to be powered by one or more backup generator blocks having a backup capacity; The operation is subsequent to at least one of the increase in the primary load or the decrease in the primary capacity, the computing device detecting that the primary load of the data center exceeds the primary capacity; a switch of the computing device connecting the one or more standby generator blocks to at least one of the one or more main generator blocks or the main load; a system wherein, following connecting the one or more standby generator blocks to at least one of the one or more main generator blocks or the primary load, the one or more standby generator blocks are connected to (a) the standby load, and (b) at least one of the one or more main generator blocks or the primary load.

9. 10. The system of claim 8, wherein connecting the one or more standby generator blocks further comprises a circuit breaker of the computing device disconnecting the standby load from the one or more standby generator blocks.

10. The system of claim 9 , wherein the auxiliary load is disconnected when a combined load including the primary load and the auxiliary load exceeds a combined capacity including the primary capacity and the auxiliary capacity.

11. 11. The system of claim 8, wherein the reserve loads are supplied with power by the one or more reserve generator blocks and a utility power connection, the utility power connection providing power to at least half of the reserve loads.

12. The system of any one of claims 8 to 10, wherein detecting that the primary load exceeds the primary capacity further comprises determining that a failure has occurred in one main generator block of the one or more main generator blocks.

13. A system according to any one of claims 8 to 10, wherein the primary load is supplied by the one or more main generator blocks or the one or more standby generator blocks with an availability of 99.999%.

14. A system according to any one of claims 8 to 10, wherein the reserve load is supplied by the one or more reserve generator blocks having an availability of 99.9%.

15. A program for causing a computer to execute the method according to any one of claims 1 to 3.

Images