Autonomous Terraforming on Cloud Infrastructure

Abstract

The process for defining a cloud service in an Infrastructure as Code (IaC) tool is designed to dynamically determine the number of clusters, availability domains, compute nodes, and / or load balancers at runtime. These values ​​are then used to determine the number of subnet levels needed for Classless Inter-Domain Routing (CIDR) slicing to generate subnets for each of the compute and load balancer nodes in the service. Because the IaC language does not provide nested loop control constructs, labels for each of the subnets may be generated and assigned using a Cartesian product of label elements. These labels may be modified by a count variable that is incremented each time a resource is replicated in the script to simulate the effect of a nested loop construct.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. patent application Ser. No. 16 / 730,656, entitled “AUTONOMOUS TERRAFORMING ON CLOUD INFRASTRUCTURES,” filed on December 30, 2019, which is incorporated by reference in its entirety. [Background technology]

[0002] background Cloud computing can provide both Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) to enable customers to securely run mission-critical enterprise applications and database workloads on a shared infrastructure. These cloud computing infrastructures may host a variety of applications, and industries are rapidly turning to microservices-based frameworks to deliver cloud-based technologies. For example, microservices projects may be hosted on a cluster of containerized projects such as Kubernetes Engine, all of which are hosted and accessible via a shared cloud infrastructure that ensures data security across tenants.

[0003] When a customer begins the process of moving their computing infrastructure to the cloud, the cloud infrastructure can create a new tenancy for that customer. Once the tenancy is created, the customer may use a console web application to create the environment building blocks needed to achieve the desired functionality of any microservices projects hosted within the cloud environment. With existing technologies, new tenants can manually create these building blocks by selecting existing technologies and adding them to their cloud footprint. Alternatively, some systems allow tenants to orchestrate the building blocks of their cloud footprint using automated scripting techniques. The automated scripts may be written in any "infrastructure as code" software tool that allows users to define and provision datacenter infrastructure using a high-level configuration language. Scripts written in these languages ​​can be executed to generate building blocks for the cloud footprint, greatly simplifying the provisioning process. However, because these scripting languages ​​are intended to represent cloud hardware / software instances, they typically do not provide features and code constructs that can be used to write robust, reusable code across many different applications. Summary of the Invention [Problem to be solved by the invention]

[0004] overview Infrastructure as Code (IaC) tools are useful for declaring resources to be provisioned as part of a service in a cloud infrastructure. However, IaC tools use a declarative language that specifies the resources to be provisioned, rather than an imperative language that executes a set of commands and control structures. Thus, an IaC script may define the specific resources and structures to be provisioned for a particular service, and the script is not easily reused for different services or provisioning tasks without significant changes to the script. [Means for solving the problem]

[0005] The embodiments described herein present a process for defining a cloud service in an IaC tool that is designed to dynamically determine the number of clusters, availability domains, compute nodes, and / or load balancers at run time. These values ​​are then used to determine the number of subnet levels required for Classless Inter-Domain Routing (CIDR) slice operations to generate subnets for each of the compute and load balancer nodes in the service. Because the IaC language does not provide nested loop control constructs, labels for each of the subnets may be generated and assigned using a Cartesian product of label elements. These labels may be altered by a count variable that is incremented each time a resource is duplicated in the script to simulate the effect of a nested loop structure.

[0006] The script can receive or determine a baseline set of parameters that may change with each service being provisioned. These parameters may be received or determined at runtime, when the script is executed, and may include the number of clusters to be provisioned as part of the service, the number of availability domains in each cluster, the number of compute nodes in each availability domain, the number of load balancers in each of the clusters, etc. The script can then determine the number of CIDR levels that ensures an adequate number of subnets for the compute nodes and / or load balancer nodes to be provisioned, along with extra subnets to provide room for future expansion. Instead of calculating these values ​​and hard-coding them into the script for each different provisioning task, these values ​​can be provided and calculated at runtime so that the script can be reused for any similar provisioning task without modification.

[0007] Assigning the computed subnets and labels to each of the declared resources would normally require a nested loop control structure in the code. However, because IaC tools typically do not provide such structures, the script may simulate nested loop behavior by computing a Cartesian product of vectors containing the components of each subnet label. For example, subnet labels generated using availability domain and cluster labels may be generated by cross-multiplying these label vectors together to generate a list of all combinations of availability domains and clusters. These labels may be further modified to include a counter value that is used to regenerate the resources declared in the script.

[0008] BRIEF DESCRIPTION OF THE DRAWINGS A further understanding of the nature and advantages of various embodiments may be realized by reference to the remaining portions of the specification and the drawings, in which like reference numerals are used to refer to like components throughout the several views. In some cases, a sub-label is associated with a reference numeral to indicate one of multiple similar components. When referring to a reference numeral without specifying an existing sub-label, the intention is to refer to all such multiple similar components. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an architectural design of a structure tenancy and cloud environment, according to some embodiments. [Diagram 2] FIG. 2 illustrates how a single network address can be “sliced” into multiple different networks, according to some embodiments. [Figure 3A] FIG. 1 illustrates a first input specifying a number of clusters that may be determined and / or received by an algorithm, according to some embodiments. [Figure 3B] FIG. 13 illustrates a second input specifying the number of availability domains that may be determined and / or received by the algorithm, according to some embodiments. [Figure 3C] FIG. 13 illustrates a third input that specifies the number of worker subnets that may be determined and / or received by the algorithm, according to some embodiments. [Figure 3D] A diagram illustrating a fourth input specifying the number of load balancer subnets that may be determined and / or received by the algorithm, according to some embodiments. [Figure 4] FIG. 13 illustrates how a Cartesian product can be used to form a single list of labels from multiple different variables, according to some embodiments. [Diagram 5] FIG. 1 illustrates how a Cartesian product may be calculated using modular arithmetic, in accordance with some embodiments. [Figure 6A] FIG. 1 illustrates a first portion of a flowchart of a method for provisioning services in a cloud infrastructure with accompanying pseudocode, according to some embodiments. [Figure 6B] 4 illustrates a continuation of the flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. [Figure 6C] 5 illustrates a continuation of the flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. [Figure 6D] 5 illustrates a continuation of the flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. [Figure 6E] 1 illustrates a completion of a flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. [Figure 7] 1 is a flowchart of a method for provisioning a variable number of subnets with corresponding labels at runtime, which may be implemented in any IaC language, according to some embodiments. [Figure 8] FIG. 1 is a simplified block diagram of a distributed system for implementing some of the embodiments. [Figure 9] FIG. 1 is a simplified block diagram of components of a system environment in which services provided by components of an embodiment of the system may be provided as cloud services. [Figure 10] FIG. 1 illustrates an exemplary computer system upon which various embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Detailed Description A cloud computing environment may include many individual building blocks, each representing a unique hardware and / or software technology. A tenant of a cloud computing infrastructure may want to design very complex cloud-based applications that can handle large volumes of web traffic and store large amounts of information for many individual users. These cloud-based applications may represent a huge complexity of interactions between different cloud technologies. However, to simplify the process of designing and provisioning complex applications, many modern cloud-based applications may be designed as microservices applications. Microservices applications use individual simple microservices as building blocks for complex technologies. When designing a large-scale application, a tenant may put together many individual microservice building blocks to form the overall functionality of the application. This process can be simplified so that large complex design tasks can be assembled by simple building blocks.

[0011] When a cloud-based application is designed, the individual hardware / software components of the design may be provisioned within the cloud environment and the application may be made available to users. Cloud provisioning is the allocation of cloud provider resources and services to a particular tenant. This disclosure focuses on the ability of a tenant or provider to automatically provision an application in a cloud service. For example, in this model, a tenant may select and assemble the building blocks of their cloud application via a web interface or portal. This provisioning model provides the user with direct control over how the application is constructed and organized within the cloud environment. This typically alleviates the need for interaction between cloud provider support services and the customer. Instead, the customer can provision the cloud application themselves, and the cloud environment can then automatically receive instructions from the customer and provision the necessary resources without the need for extensive management interaction with the process.

[0012] Despite the use of microservices and simplified building blocks during the provisioning process, the complexity of some applications can still be very daunting for users. Thus, some embodiments may use a scripting language to implement an "infrastructure as code" (IaC) environment to manage and provision cloud applications. IaC languages ​​provision computer data centers through machine-readable definition files instead of, or in addition to, physical hardware configurations or interactive configuration tools. IaC languages ​​can manage both bare metal servers and virtual machines associated with configuration resources. This allows users to model infrastructure using code, which then gives users the ability to design, realize, and deploy application infrastructure using software best practices and reusable coding techniques. IaC can be captured by a cloud computing architecture, which can then assemble the building blocks declared by the code and arrange the building blocks in a connected manner based on the code definitions.

[0013] Despite the obvious advantages associated with the use of IaC techniques, there are various limitations inherent to IaC languages. Because IaC languages ​​are intended to represent actual pieces of cloud infrastructure and their interconnectivity, they are typically designed to lack control structures that are common in true software programming languages. These inherent limitations in IaC languages ​​make it difficult for simple scripts to scale across multiple infrastructures. These languages ​​are not designed to accommodate new changes that may be required by existing and growing infrastructure. For example, classless inter-domain routing (CIDR) slicing required values ​​that were manually pre-calculated and hard-coded into scripts at the time of cloud infrastructure construction. Such approaches required human intervention to recalculate the changes, modify the code, and re-run the scripts before the code could be reused for new applications. This process is time-consuming, requires human intervention, and increases the opportunity to introduce defects into the provisioning process. In contrast to the new algorithm described in this disclosure, other techniques for making IaC languages ​​more generic and reusable have each had drawbacks that limit their effectiveness or reusability. For example, most existing solutions solve this problem by simply writing redundant code.

[0014] The embodiments described herein show new coding techniques that can be used to overcome existing limitations of IaC languages. For example, a new algorithm for CIDR slicing is developed using multiple iterators that can generate complete infrastructure blueprints of any size without requiring any manual calculations or modifications to generic code. Any changes in input variables automatically trigger a recalculation using the new algorithm, and the script may trigger a realignment of the cloud infrastructure as needed. This type of programming and execution was not possible with previous industry solutions to this technical problem. This solution uses existing IaC language features together with new coding techniques to augment the language capabilities beyond what was originally intended.

[0015] FIG. 1 illustrates an architecture design of a structure tenancy and cloud environment, according to some embodiments. Architecture 100 provides an example of an architecture that can be specified using the IaC language. When a script written in the IaC language is received and executed by a cloud infrastructure, architecture 100 can be provisioned according to the infrastructure defined by the IaC language script. Because the IaC script is a formal language that can be easily interpreted and matched to building block technologies in the cloud infrastructure, architecture 100 can be automatically provisioned by the cloud infrastructure without significant human involvement. The IaC script can lay out the individual technologies as an architecture, and the cloud infrastructure can treat the IaC script as a blueprint for provisioning each of the technologies declared in the script.

[0016] The architecture 100 of FIG. 1 is provided merely as an example and is not intended to be limiting. The components in the architecture 100 may provide examples of technology building blocks that can be declared by IaC scripts. Although a discrete number of each technology is shown in FIG. 1, it should be understood that the algorithms described below allow each of the technologies of FIG. 1 to be scaled in any number and in any combination, without limitation. The architecture 100 provides a description of how particular technologies (e.g., availability domains, load balancers, clusters, worker subnets, etc.) can be scaled by the embodiments described herein.

[0017] The architecture 100 includes multiple availability domains 102. Each of the availability domains 102 may contain some of the microservices that make up the entire application. The availability domains 102 may contain redundant services so that availability can be maintained if one of the availability domains 102 becomes unavailable. The availability domains 102 may be isolated from each other, may be fault tolerant, and may have a very low probability of failing simultaneously. In addition, the availability domains 102 may correspond to different assemblies of physical hardware located in different geographic locations, such as different regions. Alternatively, availability domains within the same region may be connected to each other by a low-latency, high-bandwidth network to build replicated systems in multiple regions for both high availability and disaster recovery.

[0018] A number of different technologies may be part of each of the availability domains 102. For example, each of the availability domains 102 may include one or more subnets 104. Each of the subnets 104 may be assigned a specific subnet address. Determining the exact number of subnets 104 may be determined in order to appropriately slice the top-level network address. This procedure, known as CIDR slicing, is described in more detail below. Each of the subnets 104 may use the assigned subnet address for inbound / outbound network access and communication between the subnets 104.

[0019] The subnets 104 may host resources found in each cluster. Many different resources may be hosted within the subnets 104, but two specific resources are shown as examples in FIG. 1. Some of the subnets 104 may include compute nodes 108. The compute nodes 108 may also be referred to as worker nodes. The compute nodes 108 perform actions associated with the corresponding worker subnet. For example, the network address of the worker subnet 104a may be used to invoke a service performed by the compute node 108a. Some of the compute nodes 108 in the cluster may be designated as a node pool 106. The subnet 104 that includes the compute nodes 108 may be referred to as a worker subnet.

[0020] In addition to holding compute nodes 108, subnets 104 may alternatively include load balancers 112. Load balancers 112 may receive and distribute traffic among the different worker subnets in each of the availability domains 102. These subnets may be referred to as load balancer subnets.

[0021] Each of the subnets 104 may also be associated with a security node 114. The security node 114 may manage security policies for each of the worker subnets 104. In some embodiments, the security node 114 may use an access list 110 to manage the systems and / or users that may access the corresponding security node 114.

[0022] Finally, the architecture 100 shown in Figure 1 may be included in a single cluster. To provide high availability of services, multiple clusters similar to the one shown in Figure 1 may be formed.

[0023] The architecture 100 may be provisioned by providing an IaC script to the cloud infrastructure. The architecture may then interpret the technologies listed in the script and automatically provision those technologies. Any IaC language may be used, such as Pulumi, Chef, Otter, Puppet SaltStack, CFEngine, DSC, Paco, etc. As an example, the embodiments described herein may use the Terraform tool 150 provided by Hashicorp. Each of these tools may use a declarative language to define the various technologies to be provisioned as part of the cloud infrastructure. As declarative languages, these IaC tools generally do not support advanced control structures that may be used in other languages. For example, the Terraform tool 150 does not support any type of looping mechanism. Instead, a resource may be instantiated a defined number of times using a count meta-argument for a resource. This prevents the tool from using any form of nested loops that may be useful for making assignments or calculations using multiple variables. Additionally, the Terraform tooling does not provide an easy way to perform CIDR slicing for a variable number of subnets that are not known at the time the code is written.

[0024] FIG. 2 illustrates how a single network address 202 can be "sliced" into multiple different networks, according to some embodiments. CIDR slicing uses address aggregation to turn a single network address into multiple addressable networks. As described above in connection with FIG. 1, each of the subnets 104 may require their own network address that exists as a subnet of the top-level network address 202. The concept of CIDR slicing subdivides the top-level network address 202 into several sub-networks based on the number of networks required. Prior to this disclosure, deriving multiple subnet addresses from the top-level network address 202 was a static operation. In other words, the user would determine the number of subnets required and then divide the top-level network address 202 accordingly. These values ​​were hard-coded into the Terraform script when the script was written and the service was designed for provisioning in a cloud infrastructure. Future scripts can be based on the existing script, but the hard-coded values ​​always need to be changed in new scripts to expand or reduce the size of the number of subnets. This required error-prone human intervention, reducing the overall usefulness of the code, and also increased the number of scripts that needed to be stored, since each script could only be used for its specific provisioning project.

[0025] The embodiments described herein solve these and other technical problems by using new algorithms and methods that conceptualize CIDR slicing as a binary tree that can be dynamically sized up and / or down as a user changes the number of availability domains, clusters, worker subnets, and / or load balancers to be provisioned. Figure 2 illustrates how a top-level network address 202 can be sliced ​​into a variable number of subnets based on level designations. For example, if a service being provisioned requires two subnets, a level 1 (211) slicing operation may be performed. If up to four subnets are used, a level 2 (212) slicing operation may be performed.

[0026] Various levels may be used based on the minimum number of subnets required. The levels 211, 212, 213, and 214 shown in FIG. 2 are based on exponential functions of base 2. For example, level 3 (213) is 3 = 8 subnets. Similarly, level 4 (214) corresponds to 2 4 = 16 subnets. To determine the number of levels needed for a slice operation, the Terraform script can determine the minimum number of subnets needed and then select a level that provides at least that many subnets. For example, if six subnets are needed for the service being provisioned, the script can dynamically determine that a level three (213) slice operation should be performed on the top network address 202. This ensures that the required number of subnets are provided along with additional subnets that may be used for future expansion.

[0027] The embodiments described herein allow the script to dynamically determine the level of the CIDR slice operation as the script is executed. As described in detail below, this does not require any hard-coded values ​​to be included in the script. Rather, the script can calculate the level required for the slice operation at runtime using the desired number of clusters, availability domains, worker subnets, load balancers, etc. In addition, as the project scales up and down, the subnet addresses can be recalculated simply by changing, for example, the number of availability domains in the project. The script can then automatically recalculate the number of levels to be used in the slice operation.

[0028] These techniques not only make the code scalable over the life of a single project, but also allow the code to be reused for future projects of any scale. The code may accept values ​​specified by the user during the service design process. For example, the user commonly specifies the number of load balancers or the number of availability domains that should be part of the project. However, the user does not specify the level of CIDR slicing that should be used to generate the appropriate number of subnet addresses. This is a low-level detail that the user would be unlikely to understand. The subnet addresses must instead be hard-coded into the Terraform script, making the code unusable for any project without the same specific architecture. The embodiments described herein allow the user to dynamically change the inputs to the script, thereby moving up and down the different levels 211, 212, 213, 214 illustrated in FIG. 2. In short, these embodiments achieve a form of automatic CIDR slicing that was previously unavailable.

[0029] The following diagrams go through different inputs that may be determined and / or provided by the script. Next, the present disclosure describes an algorithm for determining slicing levels and automatically generating corresponding subnets. FIG. 3A illustrates a first input specifying the number of clusters 302 that may be determined and / or received by the algorithm, according to some embodiments. As described above in FIG. 1, each cluster 302 may be composed of one or more availability domains, each of which may include any number of worker subnets and / or load balancers. The number of clusters 302 may be referred to herein using the variable c as shorthand in the equations described below. The number of clusters 302 may be explicitly specified by the user as an input to the script. Alternatively or additionally, the number of clusters 302 may also be determined by other values ​​and / or actions provided by the user. For example, the user may select a predefined configuration from multiple configurations available for his / her project. These configurations may correspond to a predefined number of clusters, and the script may convert the configuration to the number of clusters 302 using a lookup table, database, or the like.

[0030] FIG. 3B illustrates a second input specifying the number of availability domains 304 that may be determined and / or received by the algorithm, according to some embodiments. As described above in FIG. 1, each of the availability domains 102 may be provided to provide high availability for the service being provisioned. Functionality may be replicated across the availability domains 102 such that if one availability domain 202b becomes unavailable, the service may still be provided through the other availability domains 202a, 202c. The number of availability domains 304 may be represented using the variable a as shorthand in the equations described below. The number of availability domains 304 may be provided by a user as an input to the script. For example, a user may specify three availability domains 102 as input. Alternatively, a user may select a level of availability (e.g., high, medium, low) for the service. The script may then look up or calculate the number of availability domains 102 that corresponds to the level of availability selected for that particular project. Note that the number of availability domains 304 need not specify the total number of availability domains in the entire project. Alternatively, the number of availability domains 304 may instead be provided on a per-cluster basis, in other words, the number of availability domains 304 may represent the number of availability domains within each cluster.

[0031] For example, a configuration may be selected by a user from multiple predefined configurations. These configurations may be presented to a user as part of run-time execution of a Terraform script such that values ​​may be entered at run-time. Each configuration may be associated with a service level agreement (SLA) that specifies the level of availability required for the service. The system may derive the number of availability domains that may be required to meet the level of availability required in the SLA. Note that this same procedure may be used to determine the number of clusters, determine the number of compute nodes, and / or determine the number of load balancers.

[0032] In some embodiments, these values ​​may be dynamically received as inputs to the script at run time. This allows the script to be reused to define different services with different characteristics that would require different numbers of clusters, availability domains, compute nodes, and / or load balancers. For example, after a first service is provisioned using a script, the same script may be used to provision a second, different service. Different services may be provisioned with different subnet levels, different top-level network addresses, and / or otherwise different topologies by providing different numbers of clusters, availability domains, compute nodes, and / or load balancers as inputs to the script at run time. This allows the same script to be reused to provision different services without needing to be edited. Previously, if any of these parameters were different, a different script would have been required for each service being provisioned.

[0033] In some embodiments, these values ​​may be dynamically changed after the provisioning process has occurred. For example, a script may be executed to provision a service by receiving run-time values. After the service is provisioned, the customer may want to change the availability level, cluster location / number, and / or any other parameters. Instead of generating a new script, the same script can be executed a second time, receiving the changes to the inputs, recalculating the subnet-level numbers, and regenerating the subnets based on the new number of availability domains, clusters, compute nodes, load balancers, etc.

[0034] FIG. 3C illustrates a third input specifying the number of worker subnets 104 that may be determined and / or received by an algorithm, according to some embodiments. As described above in FIG. 1, each of the worker subnets 104 may be assigned one of the subnet addresses from the CIDR slice operation described in FIG. 2. Each worker subnet 104 may include a compute node 108. The number of worker subnets 306 may be represented using the variable w as a shorthand in the equations described below. As with the number of availability domains 304, the number of worker subnets 306 may be provided on an availability domain basis. In other words, the number of worker subnets 306 may represent the number of worker subnets in each availability domain. For example, the number of worker subnets 306 per availability domain in FIG. 3C is one worker subnet. The number of worker subnets 306 may be received and / or determined by a Terraform script. For example, a user may provide the number of worker subnets 306 per availability domain as an input to the script. Alternatively, the user may select the functions to be performed in the availability domain, and the script may automatically determine which worker subnets should be included in the design, thereby determining the number of worker subnets 306.

[0035] FIG. 3D illustrates a fourth input specifying the number of load balancer subnets 308 that may be determined and / or received by an algorithm, according to some embodiments. As described above in FIG. 1, each of the load balancer subnets may include a load balancer 112 that manages and routes traffic to the worker subnets. The number of load balancer subnets may be represented using the variable b as a shorthand in the formula described below. This number may specify the number of load balancer subnets per cluster. Typically, the user will not need to organize the load balancer subnets into specific availability domains, but instead will simply specify the total number of load balancers required in each cluster. This number may be specified as an input to the script. Alternatively, the number of load balancer subnets 308 may be determined by the script at runtime. For example, the user may specify one or more predefined configurations for the service, and the script may derive the number of load balancers required from those configurations.

[0036] Based on these inputs, the following algorithm may be used to determine the total number of subnets to be used for the services being provisioned. In one step, the algorithm may calculate the total number of worker subnets to be provisioned. The total number of worker subnets may be calculated based on the total number of clusters 302, the total number of availability domains per cluster 304, and the total number of worker subnets per availability domain 306. For example, the total number of worker subnets may be calculated using the following formula:

[0037]

number

[0038] In formula 1, w t represents the total number of worker subnets provisioned across the service.

[0039] In another step, the algorithm may calculate the total number of load balancer subnets to be provisioned. The total number of load balancer subnets may be calculated based on the total number of clusters 302 and the total number of load balancers per cluster 308. For example, the total number of load balancer subnets may be calculated using the following formula:

[0040]

number

[0041] In formula 2, b t represents the total number of load balancer subnets that should be provisioned across the service.

[0042] In another step, the algorithm may calculate a total number of subnets to be provisioned in the project. The total number of subnets may be calculated based on the total number of worker subnets and the total number of load balancer subnets. For example, the total number of subnets may be calculated using the following formula:

[0043]

number

[0044] In Equation 3, s represents the total number of subnets allocated across the entire service. The algorithm may then calculate the level to which the top network address should be sliced. As discussed above in connection with FIG. 2, the level may correspond to a base 2 exponential function that determines the number of subnet addresses that are derived from the top network address. In some embodiments, the base 2 raised to the power corresponding to the calculated number of slice levels may be equal to or greater than the total number of subnets to be allocated in the service (e.g., 2^(number of subnet levels):power with base 2 and exponent (number of subnet levels)). To determine the level, a base 2 logarithm may be calculated for the total number of subnets, and then a ceiling operation (e.g., rounding to the next highest integer) may be applied to the result of the base 2 logarithm to determine the level. For example, a particular embodiment may use the following formula to determine the number of levels:

[0045]

number

[0046] Since n∈N, the ceiling operation can be applied to Equation 7 to calculate the final level n for the slice operation.

[0047]

number

[0048] Here, the level number n may be used to perform a slice operation on the highest network address. In some embodiments, the algorithm divides the network address into two nOne may manually calculate the correct IP address for each subnet by dividing the subnet into non-overlapping subnets. Depending on the particular IaC tool used to define the architecture, there may be predefined functions that perform this operation automatically. For example, the Terraform tool provides the cidrsubnet(prefix, newbits, netnum) function that can be used to perform this mathematical operation for each subnet. This function may use the following inputs: · prefix is ​​a predetermined value derived from the highest level network address. · netnum is a value ∈ [0,2 which represents the index of one of the subnet addresses in the nth level from Figure 2. n -1]. newbits is a dynamically calculated value of n that represents the minimum value that ensures there is a sufficient CIDR block to allocate for each subnet.

[0049] At this stage, available subnets can be assigned to each of the resources described in the architecture of Figure 1. In the Terraform tool, this can be achieved using a single count meta argument for a resource. For example, a resource can be replicated once per worker subnet by inserting the line count=num_worker_subnets after the declaration of that resource in the Terraform code. It should be emphasized that IaC languages ​​and tools typically provide limited looping mechanisms that allow only a single level of resources to be replicated.

[0050] However, in practice, it is also useful to generate and / or assign labels to each of the resources based on a combination of various variables. For example, some embodiments may label each subnet according to its cluster and availability domain combination. When dealing with multiple variables that need to be iterated to generate a label for each resource, multiple looping levels may be necessary. However, as discussed above, the IaC language does not provide such nested looping structure control as a native feature of the language. Thus, in order to assign labels based on multiple variables (e.g., cluster, availability domain, etc.), each such label had to be calculated by hand and hard-coded into the script. As discussed above, hard-coding values ​​into the script limits the reusability and scalability of the design and requires human intervention whenever the design is changed.

[0051] The embodiments described herein solve this problem by collecting the various label components together into a vector of labels and performing modular arithmetic to generate a Cartesian product, which may then use the single-loop feature mimicked by the count meta argument in the Terraform tool.

[0052] FIG. 4 illustrates how a Cartesian product can be used to form a single list of labels from multiple different variables, according to some embodiments. A first vector containing a list of cluster labels 402 may include clusters referenced as production, development, staging, quality assurance, etc. A second vector containing a list of availability domains 404 may include domains referenced by numeric identifiers (e.g., AD-1, AD-2, etc.). Previously, a script might statically define all of the pairs formed by the combination of two vectors of these labels into a single hard-coded label vector, and then that single hard-coded label vector was applied to each of the subnets. However, this limits the reusability of the script. This was necessary because traditional programming languages ​​provide nested loops such that clusters 402 can be iterated over in an outer loop and availability domains 404 can be iterated over in an inner loop, but IaC tools typically do not provide such control structures.

[0053] Some embodiments approximate the nested loop control structure by using modular arithmetic as shown in FIG. 4. This allows the script to receive any number of cluster, availability domain, or other labels as input and then dynamically generate a single vector 406 of labels that can be applied using the count meta argument in the Terraform tool. Using the algorithm described below, a Cartesian product may be generated that includes the set of all ordered pairs for each of the two input label vectors. It should be emphasized that two vectors of labels are used here only as an example. Other embodiments may use more than two vector labels, and a Cartesian product may be calculated for each combination of labels to form a single label vector that may be applied without nested looping.

[0054] In FIG. 4, the Cartesian product produced a vector of labels 406 that includes each of the unique pairs of labels from the cluster 402 and the availability domain 404. FIG. 5 shows how a Cartesian product may be calculated using modular arithmetic, according to some embodiments. In this example, the Cartesian product is represented as a grid with two-dimensional coordinates. The two-dimensional coordinates represent indexes that would be used in nested loops. For example, the production cluster would loop with index 0, the development cluster would loop with index 1, the staging cluster would loop with index 2, etc. For the inner loop, each of the availability domains would loop with an index that increments from 0 to 2.

[0055] However, instead of using this nested looping process, each pair resulting from the Cartesian product can be indexed using a single index in the label result vector 406 shown in Figure 4. The single index is related to the nested indexes using modular arithmetic. For example, the index in the outer loop representing the cluster labels is related to the index by the following formula:

[0056]

number

[0057] The variable a corresponds to the number of values ​​in the input vector, which in this case is 3, and the index c corresponds to the cluster index. Similarly, the index of the inner loop representing the availability domain labels may be related to the index by the following formula:

[0058]

number

[0059] index ad corresponds to the availability domain index. 6A-6E show detailed examples of Terraform scripts implementing the above-mentioned algorithms. The flowcharts and pseudocode are provided by way of example and are not meant to be limiting. The algorithms described above may be implemented using any IaC language using the present disclosure.

[0060] FIG. 6A illustrates a first portion of a flowchart of a method for provisioning a service in a cloud infrastructure with accompanying pseudocode, according to some embodiments. The method may include receiving state variables and collecting data from the cloud infrastructure (602). As described above, some embodiments may receive as input to a script each of the variables that define the resources to be provisioned. In this example, this includes the number of availability domains, the number of worker subnets in each domain, the number of load balancer subnets for each cluster, etc. In this example, the number of worker subnets per availability domain may be assumed to be 1, and the number of load balancer subnets per cluster may be assumed to be 2. The pseudocode 601 for receiving these values ​​may include assigning each of these values ​​to a variable. The pseudocode 601 also uses the FastConnect utility, which is a network connection that is an alternative to using the public Internet to connect to cloud infrastructure and other online services.

[0061] Pseudocode 601 also includes an implementation of the algorithm described above for defining the number of CIDR levels. For example, Equation 8 above is implemented in the last line of pseudocode 601:

[0062]

number

[0063] This line of pseudocode calculates the total number of subnets needed for the load balancers and worker nodes. This total is provided as input to a base 2 logarithm function, and the result is provided to a ceiling function to determine the minimum number of CIDR levels for the service.

[0064] The method may also include provisioning 604 a gateway for the service. Pseudocode 603 shows an example of how to define a service gateway, an Internet gateway, a NAT gateway, etc. As noted above, the syntax of pseudocode 603 may be specific to the Terraform tool and therefore may be modified when working with other tools.

[0065] 6B shows a continuation of a flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. The method may further include configuring routing tables and security rules for the service (606). Pseudocode 605 shows an example for declaring a default routing table and a load balancer security list. Note that some additional security and / or routing resources may be declared in a real-world project. The example provided in pseudocode 605 is merely representative and may be used as a template for generating other security rules, security lists, and routing tables for different aspects of the service.

[0066] 6C illustrates a continuation of a flow chart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. The method may further include determining whether all worker subnets have been provisioned (608). If there are remaining worker subnets to be provisioned, the method may include calculating subnet addresses (610) using the CIDR slicing algorithm described above. Each of the subnets may then be provisioned with a corresponding CIDR block (612). This cycle may be repeated for each of the worker subnets.

[0067] As can be seen in pseudocode 614, the looping constructs shown in the flowchart cannot be represented using Terraform pseudocode. Instead, the code provides meta arguments that replicate resources a prescribed number of times. For example, the first line of the pseudocode after the resource declaration contains the following:

[0068]

number

[0069] By assigning the number of worker subnets to the count variable, pseudocode 614 can instruct the Terraform tool to create an oci_core_subnet resource for each worker subnet. Pseudocode 614 then uses cidrsubnet() above with the count.index variable used to index each of the subnets at the specified level calculated above.

[0070] Pseudocode 614 also implements the nested loop algorithm described above by computing the Cartesian product between the cluster names and the availability domain names. These are stored in the availability_domain array, which is indexed by the count.index variable. For example, the following command generates an array of labels for each of the subnets:

[0071]

number

[0072] Instead of performing a traditional nested loop, this Cartesian product generates a list of labels for the subnets. Finally, the labels can be assigned to the corresponding subnets using modular arithmetic, as shown in pseudocode 614 and described in detail above.

[0073] The process shown in FIG. 6C may be performed to allocate each of the worker subnets in a project. Although not specifically shown, the process shown in FIG. 6C may be repeated for each of the load balancer subnets. For example, a similar procedure may be followed to slice the top-level network address based on the levels calculated above, and a Cartesian product may be used to generate the list of labels. However, the netnum parameter for the cidrsubnet() function may be shifted so that the load balancer subnets are allocated to a CIDR block after the worker subnets.

[0074] 6D shows a continuation of the flow chart of a method for provisioning services in a cloud infrastructure, according to some embodiments. The method may further include determining whether enough clusters have been provisioned based on the number of clusters described above (616). If clusters remain to be provisioned, the method may determine which load balancer subnets are needed (618) and provision the correct load balancers for the clusters (620). As mentioned above, this looping procedure may be modeled in pseudocode 622 to dynamically generate a specified number of clusters using a count variable. A second portion of pseudocode 622 may assign a load balancer subnet to each of the clusters.

[0075] FIG. 6E completes a flowchart of a method for provisioning a service in a cloud infrastructure, according to some embodiments. The method may also include determining whether sufficient node pools have been provisioned (624). To the extent additional node pools need to be provisioned, the method may include dividing the subnets into node pools for allocation (626) using a ChunkList operation. In pseudocode 630, a ChunkList function from the Terraform tool can be used to divide a single list into fixed-sized chunks, thus returning a list of lists. The method may then include provisioning the node pools with corresponding worker subnets (628). In pseudocode 630, each node pool has a specified quantity of nodes per subnet described in a family map.

[0076] While the above flowchart and description are specific to the Terraform tool, the algorithm for simulating nested loops to generate and assign CIDR subnets and apply labels to the subnets may be implemented in any IaC language. FIG. 7 illustrates a flowchart of a method for provisioning a variable number of subnets at runtime along with corresponding labels, according to some embodiments, that may be implemented in any IaC language. The method may include determining (702) a number of clusters to be provisioned as part of a service in a cloud infrastructure. The method may also include determining (704) a number of availability domains in each of the clusters, determining (706) a number of compute nodes in each of the availability domains, and / or determining (708) a number of load balancers in each of the clusters. In each of these steps, these values ​​may be determined by being received as input to a script, by being stored as constants in the script, and / or by being determined by a calculation or function in the script. For example, each value may be derived or looked up based on a predefined configuration selected by a user. In some embodiments, these values ​​may be determined as described above in connection with FIGS. 3A-3B, or as shown in FIG. 6A.

[0077] The method may also include calculating (710) a number of subnet levels based on the number of clusters, the number of availability domains, the number of compute nodes, and the number of load balancers. The compute nodes may correspond to the worker subnets described above. Each of the load balancers may also be described as a load balancer subnet. The number of subnet levels may be calculated as described above in connection with Equations 2-8 and in connection with FIG. 6C. The method may further include generating (712) a number of subnets based on the number of subnet levels. This step may be performed using the cidrsubnet function described above. Alternatively or additionally, this step may be performed using a 2 nThe method may be performed by mathematically subdividing the top-level network address based on a subdivision of . The method may also include assigning (714) a plurality of subnets to the compute nodes and load balancers in the cloud infrastructure. As discussed above, each of the worker subnets and the load balancer subnets may be assigned a label based on the Cartesian product and modular arithmetic algorithms discussed above.

[0078] It should be appreciated that the particular steps illustrated in FIG. 7 provide a particular method for provisioning a variable number of subnets, at run time, with corresponding labels, according to various embodiments. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the invention may perform the steps outlined above in a different order. Additionally, individual steps illustrated in FIG. 7 may include multiple sub-steps that may be performed in various sequences appropriate to the individual step. Additionally, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0079] Each of the methods described herein may be implemented by a computer system. Each of the steps of these methods may be performed automatically by the computer system and / or may be provided with input / output involving a user. For example, a user may provide inputs for each step of a method, and each of these inputs may be responsive to a particular output that requests such input, which is generated by the computer system. Each input may be received in response to a corresponding requested output. Furthermore, an input may be received from a user as a data stream from another computer system, retrieved from a memory location, retrieved over a network, or requested from a web service. Similarly, an output may be provided to a user as a data stream to another computer system, stored in a memory location, transmitted over a network, or provided to a web service. In short, each of the steps of the methods described herein may be performed by a computer system and may include any number of inputs, outputs, and / or requests to and from the computer system, which may or may not involve a user. Steps that do not involve a user may be said to be performed automatically by the computer system without human intervention. Thus, in light of this disclosure, it will be understood that each step of each method described herein may be modified to include inputs to and outputs from a user, or may be performed automatically by a computer system without human intervention, where any decisions are made by a processor. Additionally, some embodiments of each of the methods described herein may be implemented as a set of instructions stored on a tangible, non-transitory storage medium to form a tangible software product.

[0080] 8 shows a simplified diagram of a distributed system 800 for implementing one embodiment. In the illustrated embodiment, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808 configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., over one or more networks 810. A server 812 may be communicatively coupled to remote client computing devices 802, 804, 806, and 808 over network 810.

[0081] In various embodiments, server 812 may be adapted to execute one or more services or software applications provided by one or more of the system's components. In some embodiments, these services may be provided to users of client computing devices 802, 804, 806, and / or 808 as web-based or cloud services, or under a software-as-a-service (SaaS) model. Users operating client computing devices 802, 804, 806, and / or 808 may then utilize one or more client applications to interact with server 812 to utilize the services provided by these components.

[0082] In the illustrated configuration, the software components 818, 820, and 822 of the system 800 are shown as being implemented on the server 812. In other embodiments, one or more of the components of the system 800 and / or the services provided by these components may be implemented by one or more of the client computing devices 802, 804, 806, and / or 808. A user operating a client computing device may then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 800. The illustrated embodiment is thus an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0083] The client computing devices 802, 804, 806, and / or 808 may be portable handheld devices (e.g., iPhone, cellular phone, iPad, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass head mounted display) running software such as Microsoft Windows Mobile, and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, and Internet, email, short message service (SMS), Blackberry, or other enabled communication protocols. The client computing devices may be general purpose personal computers, including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may be workstation computers running any of the various commercially available UNIX or UNIX-like operating systems, including, by way of example, various GNU / Linux operating systems such as Google Chrome OS. Alternatively, or in addition, client computing devices 802, 804, 806, and 808 may be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over network 810.

[0084] Although the example distributed system 800 is shown with four client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the server 812.

[0085] The network 810 in the distributed system 800 may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, and the like. By way of example only, the network 810 may be a local area network (LAN), such as one based on Ethernet, Token Ring, and the like. The network 810 may be a wide area network and the Internet. It may include virtual networks, including, but not limited to, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (e.g., networks operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 suite of protocols, Bluetooth, and / or any other wireless protocol); and / or any combination of these and / or other networks.

[0086] Servers 812 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configurations and / or combinations. In various embodiments, servers 812 may be adapted to execute one or more services or software applications described in the preceding disclosure. For example, servers 812 may correspond to servers for executing the processes described above in accordance with certain embodiments of the present disclosure.

[0087] Server 812 may run an operating system, including any of those mentioned above, as well as any commercially available server operating system. Server 812 may also run any of a variety of other server applications and / or mid-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA® server, a database server, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM® (International Business Machines), and the like.

[0088] In some implementations, server 812 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 802, 804, 806, and 808. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 812 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 802, 804, 806, and 808.

[0089] Distributed system 800 may also include one or more databases 814 and 816. Databases 814 and 816 may reside in a variety of locations. By way of example, one or more of databases 814 and 816 may reside on a non-transitory storage medium local to (and / or resident on) server 812. Alternatively, databases 814 and 816 may be remote from server 812 and communicate with server 812 via a network-based or dedicated connection. In one set of embodiments, databases 814 and 816 may reside within a storage area network (SAN). Similarly, any necessary files to perform the functions ascribed to server 812 may be stored locally on server 812 and / or remotely, as appropriate. In one set of embodiments, databases 814 and 816 may include relational databases, such as those provided by Oracle, adapted to store, update, and retrieve data in response to SQL-formatted commands.

[0090] 9 is a simplified block diagram of one or more components of a system environment 900 that may provide services provided by one or more components of an embodiment system as cloud services, according to one embodiment of the disclosure. In the illustrated embodiment, the system environment 900 includes one or more client computing devices 904, 906, and 908 that may be used by a user to interact with a cloud infrastructure system 902 that provides cloud services. The client computing devices may be configured to run a client application, such as a web browser, a proprietary client application (e.g., Oracle Forms), or some other application, that may be used by a user of the client computing device to interact with the cloud infrastructure system 902 to use the services provided by the cloud infrastructure system 902.

[0091] It should be understood that the illustrated cloud infrastructure system 902 may have other components than those shown. Additionally, the illustrated embodiment is only one example of a cloud infrastructure system that may incorporate embodiments of the present invention. In some other embodiments, cloud infrastructure system 902 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of components.

[0092] The client computing devices 904, 906, and 908 may be devices similar to those described above for 802, 804, 806, and 808.

[0093] Although the exemplary system environment 900 is shown with three client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the cloud infrastructure system 902.

[0094] Network 910 may facilitate communication and exchange of data between clients 904, 906, and 908 and cloud infrastructure system 902. Each network may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of commercially available protocols, including those described above for network 810.

[0095] Cloud infrastructure system 902 may include one or more computers and / or servers, which may include those described above for server 812.

[0096] In an embodiment, the services provided by a cloud infrastructure system may include a host of services made available to users of the cloud infrastructure system on demand, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, etc. The services provided by the cloud infrastructure system may be dynamically scaled to meet the needs of its users. A particular instantiation of a service provided by the cloud infrastructure system is referred to herein as a "service instance." In general, any service made available to users from a cloud service provider's system over a communications network such as the Internet is referred to as a "cloud service." Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from the customer's own on-premise servers and systems. For example, the cloud service provider's system may host an application, and a user may order and use that application on demand over a communications network such as the Internet.

[0097] In some examples, services in a computer network cloud infrastructure may include protected computer network access to storage, hosted databases, hosted web servers, software applications, or other services provided to users by cloud vendors or known in the art. For example, a service may include password-protected access to remote storage on the cloud over the Internet. As another example, a service may include a web services-based hosted relational database and a scripting language middleware engine for private use by networked developers. As another example, a service may include access to an email software application hosted on a cloud vendor's website.

[0098] In one embodiment, cloud infrastructure system 902 may include a suite of application, middleware, and database service offerings that are self-service, subscription-based, elastically scalable, reliable, highly available, and securely delivered to customers. An example of such a cloud infrastructure system is the Oracle Public Cloud offered by the present assignee.

[0099] In various embodiments, cloud infrastructure system 902 may be adapted to automatically provision, manage, and track customer subscriptions to services provided by cloud infrastructure system 902. Cloud infrastructure system 902 may provide cloud services through different deployment models. For example, the services may be provided under a public cloud model, where cloud infrastructure system 902 is owned by an organization that sells cloud services (e.g., owned by Oracle) and the services are made available to the general public or to companies in different industries. As another example, the services may be provided under a private cloud model, where cloud infrastructure system 902 operates only for a single organization and may provide services to one or more entities within that organization. Cloud services may also be provided under a community cloud model, where cloud infrastructure system 902 and the services provided by cloud infrastructure system 902 are shared by several organizations within an associated community. Cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

[0100] In some embodiments, the services offered by cloud infrastructure system 902 may include one or more services offered under a Software as a Service (SaaS) category, a Platform as a Service (PaaS) category, an Infrastructure as a Service (IaaS) category, or other category of services including hybrid services. A customer may order one or more services offered by cloud infrastructure system 902 via a subscription order. Cloud infrastructure system 902 then performs processing to provide the services in the customer's subscription order.

[0101] In some embodiments, the services provided by the cloud infrastructure system 902 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform may be configured to provide cloud services that fall under the SaaS category. For example, the SaaS platform may provide the ability to build and deliver a suite of on-demand applications on an integrated development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure to provide the SaaS services. By utilizing the services provided by the SaaS platform, customers can utilize applications that run on the cloud infrastructure system. Customers can obtain application services without having to purchase separate licenses and support. A variety of different SaaS services may be provided. Examples include, but are not limited to, services that provide solutions for sales performance management, enterprise integration, and business flexibility for large organizations.

[0102] In some embodiments, platform services may be provided by the cloud infrastructure system through a PaaS platform. The PaaS platform may be configured to provide cloud services that fall under the PaaS category. Examples of platform services may include, but are not limited to, services that allow organizations (such as Oracle) to integrate existing applications on a shared common architecture, and the ability to build new applications that leverage the shared services provided by the platform. The SaaS platform may manage and control the underlying software and infrastructure to provide the SaaS services. Customers can obtain the PaaS services provided by the cloud infrastructure system without the customer having to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), and the like.

[0103] By utilizing the services offered by the PaaS platform, customers can adopt programming languages ​​and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services offered by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware services), and Java cloud services. In one embodiment, the database cloud services may support a shared services deployment model that allows organizations to pool database resources and provide database-as-a-service to customers in the form of a database cloud. The middleware cloud services may provide a platform for customers to develop and deploy various business applications, and the Java cloud services may provide a platform for customers to deploy Java applications in the cloud infrastructure system.

[0104] A variety of different infrastructure services may be provided by the IaaS platform in a cloud infrastructure system that facilitates management and control of underlying computing resources, such as storage, network, and other fundamental computing resources, for customers that use the services provided by the SaaS and PaaS platforms.

[0105] In an embodiment, cloud infrastructure system 902 may also include infrastructure resources 930 for providing resources used to provide various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources 930 may include a pre-integrated and optimized combination of hardware, such as servers, storage, and networking resources to run the services offered by the PaaS and SaaS platforms.

[0106] In some embodiments, resources in cloud infrastructure system 902 may be shared by multiple users and dynamically reallocated per demand. In addition, resources may be allocated to users in different time periods. For example, cloud infrastructure system 930 may allow a first set of users in a first time period to utilize resources in the cloud infrastructure system for a specified number of hours, and then reallocate the same resources to another set of users in a different time period, thereby maximizing utilization of the resources.

[0107] In an embodiment, several internal shared services 932 may be provided that are shared by different components or modules of cloud infrastructure system 902 and by services provided by cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0108] In particular embodiments, cloud infrastructure system 902 may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one embodiment, cloud management functions may include capabilities for provisioning, managing, tracking customer subscriptions received by cloud infrastructure system 902, and the like.

[0109] In one embodiment, as shown, the cloud management functionality may be provided by one or more modules, such as an order management module 920, an order orchestration module 922, an order provisioning module 924, an order management and monitoring module 926, and an identity management module 928. These modules may include or be provided using one or more computers and / or servers, which may be general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0110] In example operation 934, a customer using a client device, such as client device 904, 906, or 908, may interact with cloud infrastructure system 902 by requesting one or more services offered by cloud infrastructure system 902 and placing an order for a subscription to one or more services offered by cloud infrastructure system 902. In an embodiment, the customer may access cloud user interfaces (UIs), cloud UI 912, cloud UI 914, and / or cloud UI 916, and place the subscription order via these UIs. Order information received by cloud infrastructure system 902 in response to the customer placing the order may include information identifying the customer and the one or more services offered by cloud infrastructure system 902 to which the customer wishes to subscribe.

[0111] After an order is placed by a customer, the order information is received via the cloud UI 912, 914 and / or 916.

[0112] In operation 936, the order is stored in order database 918. Order database 918 can be one of several databases operated by cloud infrastructure system 918 and in conjunction with other system elements.

[0113] At operation 938, the order information is forwarded to the order management module 920. In some examples, the order management module 920 may be configured to perform billing and accounting functions related to the order, such as validating the order and booking the order once verified.

[0114] At operation 940, information regarding the order is communicated to the order orchestration module 922. The order orchestration module 922 may utilize the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some examples, the order orchestration module 922 may use the services of the order provisioning module 924 to orchestrate the provisioning of resources to support the subscribed services.

[0115] In certain embodiments, the order orchestration module 922 enables management of business processes associated with each order and applies business logic to determine whether the order should proceed to provisioning. Upon receiving a new subscription order, at operation 942, the order orchestration module 922 sends a request to the order provisioning module 924 to allocate resources and configure the resources needed to fulfill the subscription order. The order provisioning module 924 enables the allocation of resources for the services ordered by the customer. The order provisioning module 924 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 900 and the physical implementation layer used to provision resources to provide the requested services. Thus, the order orchestration module 922 may be insulated from implementation details, such as whether services and resources are actually provisioned on the fly or pre-provisioned and allocated / allocated only on demand.

[0116] In operation 944 , once the services and resources have been provisioned, a notification of the rendered services may be sent by the order provisioning module 924 of the cloud infrastructure system 902 to the customer on the client device 904 , 906 and / or 908 .

[0117] At operation 946, the customer's subscription order may be managed and tracked by the order management and monitoring module 926. In some cases, the order management and monitoring module 926 may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system up and down time.

[0118] In an embodiment, cloud infrastructure system 900 may include an identity management module 928. Identity management module 928 may be configured to provide identity services, such as access management and authorization services in cloud infrastructure system 900. In some embodiments, identity management module 928 may control information about customers who wish to use services provided by cloud infrastructure system 902. Such information may include information authenticating the identities of such customers and information describing what actions those customers are permitted to perform on various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). Identity management module 928 may also include management of descriptive information about each customer, as well as how and who can access and modify that descriptive information.

[0119] 10 illustrates an exemplary computer system 1000 upon which various embodiments of the present invention may be implemented. The system 1000 may be used to implement any of the computer systems described above. As shown in the figure, the computer system 1000 includes a processing unit 1004 that communicates with several peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and a system memory 1010.

[0120] Bus subsystem 1002 provides a mechanism for allowing the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0121] The processing unit 1004 may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers) and controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core processors or multi-core processors. In particular embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, with a single-core processor or a multi-core processor included in each processing unit. In other embodiments, the processing unit 1004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0122] In various embodiments, the processing unit 1004 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed may reside in the processor 1004 and / or in the storage subsystem 1018. Through suitable programming, the processor 1004 may provide various functionality as described above. The computer system 1000 may further include a processing acceleration unit 1006, which may include a digital signal processor (DSP), special purpose processor, etc.

[0123] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include motion sensing and / or gesture recognition devices such as a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices such as a Google Glass® blink detector that detects eye movements from a user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0124] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, medical ultrasound machines, etc. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, etc.

[0125] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0126] Computer system 1000 may also include a storage subsystem 1018 that includes software elements currently shown as located in system memory 1010. System memory 1010 may store program instructions loadable and executable on processing unit 1004, as well as data generated during the execution of these programs.

[0127] Depending on the configuration and type of computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 1004. In some implementations, the system memory 1010 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1000, such as during start-up, may typically be stored in ROM. By way of example and not limitation, the system memory 1010 also illustrates application programs 1012, program data 1014, and an operating system 1016, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like. By way of example, operating system 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS operating systems.

[0128] The storage subsystem 1018 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the functionality described above may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with the present invention.

[0129] Storage subsystem 1000 may also include a computer readable storage medium reader 1020 that may be further connected to a computer readable storage medium 1022. Along with, and optionally in combination with, the system memory 1010, computer readable storage medium 1022 may collectively represent remote, local, fixed, and / or removable storage devices plus storage media for containing, storing, transmitting, and retrieving computer readable information on a temporary and / or more permanent basis.

[0130] The computer readable storage medium 1022 containing the code or portions of code may also include any suitable medium known or used in the art, including, but not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media implemented in any manner or technology for the storage and / or transmission of information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable medium. This may also include non-tangible computer readable media, such as a data signal, data transmission, or any other medium that can be used to transmit the desired information and that can be accessed by the computing system 1000.

[0131] By way of example, the computer readable storage medium 1022 may include a hard disk drive that reads and writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads and writes to a removable non-volatile magnetic disk, an optical disk drive that reads and writes to a removable non-volatile optical disk, such as a CD ROM, a DVD, and a Blu-Ray® disk, or other optical media. The computer readable storage medium 1022 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage medium 1022 may also include a flash memory-based SSD, an enterprise flash drive, a solid state drive (SSD) based on a non-volatile memory such as a solid state ROM, an SSD based on a volatile memory such as a solid state RAM, a dynamic RAM, a static RAM, a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD that uses a combination of DRAM and a flash memory-based SSD. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1000.

[0132] The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for sending and receiving data between other systems and the computer system 1000. For example, the communications subsystem 1024 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1024 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1024 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0133] In some embodiments, the communications subsystem 1024 may also receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. on behalf of one or more users who may be using the computer system 1000.

[0134] As an example, the communications subsystem 1024 may be configured to receive data feeds 1026 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0135] In addition, the communications subsystem 1024 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1028 of real-time events and / or event updates 1030, which may be continuous or infinite in nature with no explicit termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0136] The communications subsystem 1024 may also be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1000.

[0137] The computer system 1000 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0138] Due to the ever-changing nature of computers and networks, the description of the computer system 1000 shown in the figures is intended only as a specific example. Many other configurations are possible having more or fewer components than the system depicted in the figures. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other aspects and / or methods for implementing various embodiments.

[0139] In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it will be apparent to one skilled in the art that embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.

[0140] The foregoing description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the foregoing description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the claims.

[0141] In the above description, specific details are given to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0142] It should also be noted that particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process is terminated when the operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. Where a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0143] The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing, or carrying instructions and / or data. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0144] Furthermore, the embodiments may be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. The necessary tasks may be executed by a processor.

[0145] In the foregoing specification, aspects of the invention have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. The various features and aspects of the invention described above may be used individually or together. Moreover, the embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.

[0146] Furthermore, for purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described. It should also be understood that the above methods may be performed by hardware components or embodied in a sequence of machine-executable instructions that, when used, cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with such instructions, to perform the above methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or the like, or any other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the method may be performed by a combination of hardware and software.

Claims

1. 1. A program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: executing a provisioning script in an infrastructure as code (IaC) programming language, the script being executed first before a service is provisioned in the cloud infrastructure as a result of the script, the execution of the script comprising: determining, upon execution of the script, a number of clusters to be provisioned as part of the service in the cloud infrastructure; determining, upon execution of the script, a number of availability domains for each of the clusters to be provisioned in the cloud infrastructure; determining, upon execution of the script, a number of compute nodes in each of the availability domains to be provisioned in the cloud infrastructure; and determining, upon execution of the script, a number of load balancers for each of the clusters to be provisioned in the cloud infrastructure; Calculating a number of subnet levels based on a number of the clusters, the number of the availability domains, the number of the compute nodes, and the number of the load balancers upon execution of the script; generating a number of subnets based on the number of subnet levels upon execution of the script; and assigning the plurality of subnets to the compute nodes and the load balancers in the cloud infrastructure upon execution of the script.

2. The method of claim 1 , wherein the operations further comprise causing each of the compute nodes and each of the load balancers to be provisioned with the multiple subnets in the cloud infrastructure.

3. The operation further comprises: receiving a change to the number of availability domains, the number of clusters, the number of compute nodes, or the number of load balancers; recalculating the number of subnet levels based on the changes; and and regenerating the plurality of subnets based on the number of subnet levels.

4. The operation further comprises: determining a second number of clusters, a second number of availability domains, a second number of compute nodes, and a second number of load balancers to be provisioned in the cloud infrastructure as part of a second service, the second service being different from the service, the operations further comprising: calculating a second number of subnet levels based on the second number of clusters, the second number of availability domains, the second number of compute nodes, and the second number of load balancers, the second number of subnet levels being different from the number of subnet levels, the operations further comprising: The program of claim 1, further comprising generating a second plurality of subnets based on the second number of subnet levels.

5. 5. The program of claim 1, wherein the number of the plurality of subnets is equal to base 2^ (the number of subnet levels).

6. Determining the number of availability domains includes: receiving a selection of a configuration from a plurality of predefined configurations for the service; and determining a service level agreement (SLA) for the configuration, the SLA defining the number of availability domains.

7. Determining the number of availability domains includes: The program of claim 1 , wherein the instructions, when executed by the one or more processors, comprise receiving the number of availability domains as a runtime input.

8. The program according to any one of claims 1 to 7, wherein an availability domain in the availability domain includes at least one of the compute nodes and at least one of the load balancers.

9. generating the plurality of subnets based on the number of subnet levels, 9. The program of claim 1, further comprising executing a function for performing Classless Inter-Domain Routing (CIDR) slicing of a top-level network address.

10. 10. The program of claim 9, wherein the function accepts the number of subnet levels as a parameter such that the number of subnet levels can be calculated at run time.

11. The method of claim 1, wherein the operations further comprise: assigning a plurality of labels to each of the plurality of subnets.

12. The method of claim 11 , wherein the plurality of labels is based on a combination of the availability domain and the cluster.

13. 13. The method of claim 12, wherein the operations further comprise generating the combinations of the availability domains and the clusters by computing a Cartesian product.

14. 14. The method of claim 13, wherein computing the Cartesian product comprises generating a vector cross product between the availability domain and the cluster.

15. 15. The method of claim 14, wherein an index in the plurality of labels is related to an index in the cluster by dividing the index in the plurality of labels by a number of clusters.

16. 15. The program of claim 14, wherein the indexes at the plurality of labels are related to indexes at the availability domains by performing a modular division of the indexes at the plurality of labels by the number of availability domains.

17. The method of claim 12 , wherein the plurality of labels includes an index that is changed by a counter that is incremented each time a compute node or a load balancer is declared.

18. The method of claim 12 , wherein the plurality of labels are generated without using a nested loop structure.

19. 1. A system comprising: one or more processors; A system comprising one or more memory devices storing the program according to any one of claims 1 to 18.

20. 1. A method for executing a provisioning script with runtime adjustment resource declarations, the method comprising: executing a provisioning script in an infrastructure as code (IaC) programming language, the script being executed first before a service is provisioned in the cloud infrastructure as a result of the script, the execution of the script comprising: determining, upon execution of the script, a number of clusters to be provisioned as part of the service in the cloud infrastructure; determining, upon execution of the script, a number of availability domains for each of the clusters to be provisioned in the cloud infrastructure; determining, upon execution of the script, a number of compute nodes in each of the availability domains to be provisioned in the cloud infrastructure; and determining, upon execution of the script, a number of load balancers for each of the clusters to be provisioned in the cloud infrastructure; Calculating a number of subnet levels based on a number of the clusters, the number of the availability domains, the number of the compute nodes, and the number of the load balancers upon execution of the script; generating a number of subnets based on the number of subnet levels upon execution of the script; and assigning the plurality of subnets to the compute nodes and the load balancers in the cloud infrastructure upon execution of the script.

Images