Autonomous terraforming on cloud infrastructure

Abstract

To present a process for defining cloud services in an Infrastructure as Code (IaC) tool.SOLUTION: A method dynamically determines the number of clusters, availability domains, compute nodes, and load balancers to define cloud services in an IaC tool. These values are used for determining the number of subnet levels required for Classless Inter-Domain Routing (CIDR) slicing operations to generate subnets for the compute nodes and load balancer nodes in the service. Because IaC languages do not provide nested loop control constructs, labels for each of the subnets are generated and assigned using a Cartesian product of label elements. These labels are modified by a count variable that is incremented each time a resource is duplicated in a script to simulate the effect of a nested loop construct.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit and priority of U.S. Patent Application No. 16 / 730,656, titled "AUTONOMOUS TERRAFORMING ON CLOUD INFRASTRUCTURES", filed on December 30, 2019, the entire disclosure of which is hereby incorporated by reference herein.

Background Art

[0002] Background Cloud computing can provide both Infrastructure - as - a - Service (IaaS) and Platform - as - a - Service (PaaS) to enable customers to securely run mission - critical enterprise applications and database workloads on a shared infrastructure. These cloud computing infrastructures may host various applications, and the industry is rapidly moving towards a microservices - based framework for providing cloud - based technologies. For example, a microservices project may be hosted in a cluster of containerized projects such as Kubernetes Engine, and all of them are hosted and accessible via a shared cloud infrastructure that guarantees data security between tenants.

[0003] ​When a customer starts the process of moving their computing infrastructure to the cloud, the cloud infrastructure can create a new tenancy for that customer. Once the tenancy is created, the customer may use the console web application to create the necessary infrastructure building blocks to achieve the desired functionality of any microservices project hosted within the cloud environment. With existing technology, a new tenant can manually create these building blocks by selecting existing technologies and adding them to their cloud footprint. Alternatively, some systems enable the tenant to orchestrate the building blocks of their cloud footprint using automated scripting techniques. The automated scripts may be written in any "Infrastructure as Code" software tool that enables the user to define and provision data center infrastructure using a high-level configuration language. Scripts written in these languages can be executed to generate the building blocks for the cloud footprint and can significantly simplify the provisioning process. However, since these scripting languages are intended to represent cloud hardware / software instances, they typically do not provide the features and code constructs that can be used to write robust and reusable code across many different applications. Summary of the Invention Problems to be Solved by the Invention

[0004] Overview Infrastructure as Code (IaC) tools are useful for declaring resources that are provisioned as part of a service in cloud infrastructure. However, IaC tools use a declarative language that specifies the resources to be provisioned rather than an imperative language that executes a set of commands and control structures. Thus, an IaC script can define the specific resources and structures to be provisioned for a particular service, and the script cannot be easily reused without significant changes to the script for different services or provisioning tasks. without significant changes to the script for different services or provisioning tasks.

Means for Solving the Problem

[0005] Embodiments described herein present a process for defining cloud services in an IaC tool, where the IaC tool is designed to dynamically determine, at runtime, the number of clusters, availability domains, compute nodes, and / or load balancers. These values are then used to determine the number of subnet levels required for Classless Inter-Domain Routing (CIDR) slice operations to generate subnets for each of the compute nodes and load balancer nodes in the service. Since the IaC language does not provide nested loop control structures, labels for each of the subnets can be generated and assigned using the Cartesian product of label elements. These labels can be changed by a count variable that is incremented each time a resource is replicated in the script to simulate the effect of a nested loop structure.

[0006] The script can receive or determine a baseline set of parameters that may vary with each provisioned service. These parameters can be received or determined at runtime, when the script is executed, and may include the number of clusters to be provisioned as part of the service, the number of availability domains in each cluster, the number of compute nodes in each availability domain, the number of load balancers in each of the clusters, and so on. The script can then determine the CIDR-level number that guarantees an appropriate number of subnets for the compute nodes and / or load balancer nodes to be provisioned, along with extra subnets to provide room for future expansion. Instead of hard-coding these values into the script for each different provisioning task, these values can be provided and calculated at runtime so that the script can be reused for any similar provisioning task without modification.

[0007] Assigning the calculated subnets and labels to each of the declared resources will typically require a nested loop control structure within its code. However, since IaC tools typically do not provide such a structure, the script can simulate the behavior of a nested loop by calculating the Cartesian product of vectors containing the components of each subnet label. For example, subnet labels generated using availability domain and cluster labels can be generated by cross-multiplying these label vectors to produce a list of all combinations of availability domains and clusters. These labels can be further modified to include counter values used to reproduce the resources declared within the script.

[0008] Brief Description of the Drawings A further understanding of the nature and advantages of the various embodiments can be realized by reference to the remainder of the specification and the drawings, wherein like reference numerals are used throughout the several drawings to refer to similar components. In some cases, a lower label is associated with the reference numeral to indicate one of a plurality of similar components. When referring to a reference numeral without designating an existing lower label, it is intended to refer to all such plurality of similar components.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 7

Figure 8

Figure 9

Figure 10

[0010] Detailed Description A cloud computing environment may include a number of individual building blocks each representing its own hardware and / or software technology. Tenants of a cloud computing infrastructure may wish to design very complex cloud-based applications that can handle large amounts of web traffic and store large amounts of information for a number of individual users. These cloud-based applications can represent the vast complexity of interactions between different cloud technologies. However, to simplify the process of designing and provisioning complex applications, many modern cloud-based applications can be designed as microservices applications. Microservices applications use individual simple microservices as building blocks for complex technologies. When designing large-scale applications, a tenant may bring together many individual microservices building blocks to form the overall functionality of the application. This process can be simplified so that large and complex design tasks can be assembled by simple building blocks.

[0011] When a cloud-based application is designed, the individual hardware / software components of the design may be provisioned within the cloud environment and the application may be made available to users. Cloud provisioning is the allocation of cloud provider resources and services to a particular tenant. This disclosure focuses on the ability of a tenant or provider to automatically provision applications in cloud services. For example, in this model, a tenant may select and assemble the building blocks of its cloud application via a web interface or portal. This provisioning model provides the user with direct control over how the application is built and orchestrated within the cloud environment. This typically reduces the need for interaction between the cloud provider support service and the customer. Instead, the customer can provision its own cloud application and then the cloud environment can automatically receive instructions from the customer and provision the required resources without the need for extensive management interaction with the process.

[0012] Despite the use of microservices and simplified building blocks during the provisioning process, the complexity of some applications can still be very difficult for users. Thus, some embodiments may implement an "Infrastructure as Code" (IaC) environment using a scripting language to manage and provision cloud applications. IaC languages provision computer data centers through machine-readable definition files instead of, or in addition to, physical hardware configurations or interactive configuration tools. IaC languages can manage both bare metal servers and virtual machines associated with configured resources. This enables users to model infrastructure using code, which in turn gives users the ability to design, implement, and deploy application infrastructure using software best practices and reusable coding techniques. IaC can be incorporated by cloud computing architectures, which can then assemble the building blocks declared by the code and place the building blocks in a connected manner based on the code definition.

[0013] Despite the obvious advantages associated with the use of IaC technologies, there are various limitations inherent to IaC languages. Since IaC languages are intended to represent the actual parts of cloud infrastructure and their interconnectivity, they are typically designed to lack control structures that are common in true software programming languages. These inherent limitations in IaC languages make it difficult for simple scripts to scale across multiple infrastructures. These languages are not designed to adapt to new changes that may be required by existing and growing infrastructures. For example, Classless Inter-Domain Routing (CIDR) slicing required values that were manually pre-computed and hard-coded into the script at the time of building cloud infrastructure. Such an approach required human intervention to recalculate changes, modify the code, and re-run the script before the code could be reused for new applications. This process was time-consuming, required human intervention, and increased the opportunity to introduce defects into the provisioning process. In contrast to the new algorithms described in this disclosure, other techniques for making IaC languages more general-purpose and reusable each had drawbacks that limited their effectiveness or reusability. For example, most existing solutions simply solve this problem by writing redundant code.

[0014] The embodiments described herein show new coding techniques that can be used to overcome existing limitations of IaC languages. For example, generating a complete infrastructure blueprint of any size that requires no manual calculation or changes to general-purpose code A new algorithm for CIDR slicing is being developed using multiple iterators that can be used. Any change to the input variables automatically triggers a recalculation using the new algorithm, and the script may trigger a realignment of the cloud infrastructure as needed. This type of programming and execution was not possible with previous industry solutions to this technical problem. This solution uses existing IaC language capabilities along with new coding techniques to enhance the language capabilities beyond what they were originally intended for.

[0015] Figure 1 illustrates an architectural design of a structural tenancy and cloud environment according to some embodiments. Architecture 100 provides an example of an architecture that can be specified using an IaC language. When a script written in the IaC language is received and executed by a cloud infrastructure, architecture 100 can be provisioned according to the infrastructure defined by the IaC language script. Since the IaC script is a formal language that can be easily interpreted and can match building block technology in a cloud infrastructure, architecture 100 can be automatically provisioned by the cloud infrastructure without significant human involvement. The IaC script can lay out individual technologies as an architecture, and the cloud infrastructure can treat the IaC script as a blueprint for provisioning each of the technologies declared in the script.

[0016] The architecture 100 of FIG. 1 is provided by way of example only and is not intended to be limiting. The components in architecture 100 may provide examples of technical building blocks that can be declared by IaC scripts. Although each of the discrete number of technologies is shown in FIG. 1, it should be understood that the algorithms described below enable each of the technologies in FIG. 1 to be scaled in any number and any combination without limitation. Architecture 100 provides a description of how specific technologies (e.g., availability domains, load balancers, clusters, worker subnets, etc.) can be scaled by the embodiments described herein.

[0017] The architecture 100 includes a plurality of availability domains 102. Each of the availability domains 102 may include a portion of the microservices that build the entire application. The availability domains 102 may include redundant services so that availability can be maintained if one of the availability domains 102 becomes unavailable. The availability domains 102 may be isolated from each other, fault tolerant, and have a very low probability of failing simultaneously. Additionally, the availability domains 102 may correspond to different assemblies of physical hardware located in different geographical locations such as different regions. Alternatively, availability domains within the same region may be interconnected by a low-latency high-bandwidth network to build a system replicated in multiple regions for both high availability and disaster recovery.

[0018] Several different technologies may be part of each of the availability domains 102. For example, each of the availability domains 102 may include one or more subnets 104. Each of the subnets 104 may be assigned a specific subnet address. Determining the exact number of subnets 104 may be determined to appropriately slice the top-level network address. This procedure, known as CIDR slicing, is described in more detail below. Each of the subnets 104 may use the assigned subnet address for inbound / outbound network access and communication between the subnets 104.

[0019] The subnets 104 may host the resources found in each cluster. Many different resources may be hosted within the subnets 104, but two specific resources are shown as examples in FIG. 1. Some of the subnets 104 may include compute nodes 108. Compute nodes 108 may also be referred to as worker nodes. Compute nodes 108 execute actions associated with the corresponding worker subnet. For example, the network address of worker subnet 104a may be used to call a service executed by compute node 108a. Some of the compute nodes 108 within the cluster may be designated as a node pool 106. Subnets 104 that include compute nodes 108 may be referred to as worker subnets.

[0020] In addition to holding compute nodes 108, the subnets 104 may alternatively include load balancers 112. Load balancers 112 may receive and distribute traffic between different worker subnets in each of the availability domains 102. These subnets may be referred to as load balancer subnets.

[0021] Each of the subnets 104 may also be associated with a security node 114. The security node 114 may manage the security policy of each of the worker subnets 104. In some embodiments, the security node 114 may use an access list 110 to manage systems and / or users that may access the corresponding security node 114.

[0022] Finally, the architecture 100 shown in FIG. 1 may be included in a single cluster. To achieve high availability of the services, multiple clusters identical to the cluster shown in FIG. 1 may be formed.

[0023] The architecture 100 may be provisioned by providing IaC scripts to the cloud infrastructure. The structure may then interpret the techniques enumerated in the script and automatically provision those techniques. Any IaC language such as Pulumi, Chef, Otter, Puppet SaltStack, CFEngine, DSC, Paco, etc. may be used. As an example, the embodiments described herein may use the Terraform tool 150 provided by Hashicorp. Each of these tools may use a declarative language to define the various techniques to be provisioned as part of the cloud infrastructure. As declarative languages, these IaC tools generally do not support advanced control structures that may be used in other languages. For example, the Terraform tool 150 does not support any type of looping mechanism. Instead, a resource may be instantiated a defined number of times using a count meta-argument for a resource. This prevents the tool from using any form of nested loop that may be useful for performing assignments or calculations using multiple variables. Further, the Terraform tool does not support variables that are not known at the time the code is written. No type of looping mechanism is supported. Instead, a resource may be instantiated a defined number of times using a count meta-argument for a resource. This prevents the tool from using any form of nested loop that may be useful for performing assignments or calculations using multiple variables. Further, the Terraform tool does not support variables that are not known at the time the code is written. It does not provide an easy way to perform CIDR slicing on the subnet.

[0024] Figure 2 shows how a single network address 202 can be "sliced" into multiple different networks according to some embodiments. CIDR slicing uses address aggregation to change a single network address into multiple addressable networks. As described above in connection with FIG. 1, each of the subnets 104 may require its own network address that exists as a subnet of the top-level network address 202. The concept of CIDR slicing subdivides the top-level network address 202 into several subnetwork addresses based on the number of networks required. Prior to the present disclosure, deriving multiple subnetwork addresses from the top-level network address 202 was a static operation. In other words, the user would determine the number of subnets required and then divide the top-level network address 202 accordingly. These values were hard-coded into the Terraform script when the script was written and designed for provisioning in the cloud infrastructure. Future scripts can be based on existing scripts, but the hard-coded values always need to be changed in the new script to expand or shrink the size of the number of subnets. This required error-prone human intervention and reduced the overall usefulness of the code. Also, since each script can only be used for the corresponding specific provisioning project, the number of scripts that need to be saved has also increased.

[0025] The embodiments described herein solve these and other technical problems by using a new algorithm and method that conceptualize CIDR slicing as a binary tree that can be dynamically sized up and / or down as the user changes the number of availability domains, clusters, worker subnets, and / or load balancers to be provisioned. Figure 2 shows how the top-level network address 202 can be sliced into a variable number of subnets based on level designations. For example, if the provisioned service requires two subnets, a slice operation at level 1 (211) may be performed. If up to four subnets are used, a slice operation at level 2 (212) may be performed.

[0026] Based on the minimum number of subnets required, various levels may be used. The levels 211, 212, 213, 214 shown in Figure 2 are based on a base-2 exponential function. For example, level 3 (213) corresponds to 2 3 = 8 subnets. Similarly, level 4 (214) corresponds to 2 4 = 16 subnets. To determine the level number required for the slice operation, the Terraform script can determine the minimum number of subnets required and then select the level that provides at least that number of subnets. For example, if six subnets are required for the provisioned service, the script can dynamically determine that a slice operation at level 3 (213) should be performed on the top-level network address 202. This ensures that the required number of subnets is provided along with additional subnets that may be used for future expansion.

[0027] The embodiments described in this specification enable a script to dynamically determine the level of CIDR slicing operations when the script is executed. As will be described in detail below, this does not require any hard-coded values to be included in the script. Rather, the script can calculate the levels required for slicing operations at runtime using the desired number of clusters, availability domains, worker subnets, load balancers, etc. In addition, as the project scales up and down, the subnet addresses can be recalculated simply by, for example, changing the number of availability domains within the project. The script can then automatically recalculate the number of levels used in the slicing operation.

[0028] These techniques not only make the code scalable over the lifetime of a single project, but also enable the code to be reused for future projects of any size. The code may accept values specified by the user during the service design process. For example, the user commonly specifies the number of load balancers or the number of availability domains that should be part of the project. However, the user does not specify the level of CIDR slicing operations that should be used to generate the appropriate number of subnet addresses. This is a low-level detail that would likely be less understood by the user. The subnet addresses would instead have to be hard-coded into the Terraform script, making the code inaccessible for use with any project without the same specific architecture. The embodiments described in this specification enable the user to dynamically change the input to the script, thereby moving the different levels 211, 212, 213, 214 illustrated in FIG. 2 up and down. In short, these embodiments achieve a form of automatic CIDR slicing that was not previously available.

[0029] The following figures pass through different inputs that may be determined and / or provided by a script. Next, the present disclosure describes an algorithm for determining a slicing level and automatically generating corresponding subnets. FIG. 3A shows a first input specifying the number 302 of clusters that may be determined and / or received by the algorithm, according to some embodiments. As described above with reference to FIG. 1, each cluster 302 may be composed of one or more availability domains, and each of the availability domains may include any number of worker subnets and / or load balancers. The number 302 of clusters may be referred to herein as the variable c as an abbreviated expression in the equations described below. The number 302 of clusters may be explicitly specified by the user as an input to the script. Alternatively or additionally, the number 302 of clusters may also be determined by other values and / or operations provided by the user. For example, the user may select a predetermined configuration from a plurality of configurations available for their project. These configurations may correspond to a predetermined number of clusters, and the script may convert the configuration to the number 302 of clusters using a lookup table, a database, etc.

[0030] Figure 3B shows a second input that specifies the number of availability domains 304 that may be determined and / or received by an algorithm, according to some embodiments. As described above with respect to FIG. 1, each of the availability domains 102 may be provided to provide high availability for the service being provisioned. The functionality may be replicated across the availability domains 102 so that the service may still be provided through other availability domains 202a, 202c if one availability domain 202b becomes unavailable. The number of availability domains 304 may be represented as the variable a as an abbreviation in the equations described below. The number of availability domains 304 may be provided by a user as an input to the script. For example, the user may specify three availability domains 102 as an input. Alternatively, the user may select a level of availability (e.g., high, medium, low) for the service. The script may then search or calculate the number of availability domains 102 corresponding to the selected level of availability for that particular project. Note that the number of availability domains 304 need not specify the total number of availability domains across the entire project. Instead, the number of availability domains 304 may instead be provided on a per-cluster basis. In other words, the number of availability domains 304 may represent the number of availability domains within each cluster.

[0031] For example, a configuration may be selected by a user from a plurality of predefined configurations. These configurations may be presented to the user as part of the runtime execution of the Terraform script so that values may be input at runtime. Each configuration may be associated with a service level agreement (SLA) that specifies the level of availability required for the service. The system may derive the number of availability domains required to meet the level of availability required in the SLA. Note that this same procedure may be used to determine the number of clusters, the number of compute nodes, and / or the number of load balancers.

[0032] ​ In some embodiments, these values may be dynamically received as input to the script at runtime. This enables reusing the script to define different services that would require different numbers of clusters, availability domains, compute nodes, and / or load balancers, and / or have different characteristics. For example, after provisioning a first service using the script, the same script may be used to provision a second, different service. By providing different numbers of clusters, availability domains, compute nodes, and / or load balancers as input to the script at runtime, different services with different subnetwork levels, different top-level network addresses, and / or other different topologies may be provisioned. This allows the same script to be reused to provision different services without the need to be edited. Previously, if any of these parameters were different, different scripts would have been required for each provisioned service.

[0033] In some embodiments, these values may be dynamically changed after the provisioning process has occurred. For example, the script may be executed to provision a service by receiving runtime values. After the service is provided, the customer may wish to change the level of availability, cluster location / number, and / or any other parameter. Instead of generating a new script, the second execution of the same script can be received with changes to the input, recalculate the number of subnetwork levels, and regenerate the subnetwork based on the new number of availability domains, clusters, compute nodes, load balancers, etc.

[0034] Figure 3C shows a third input that, according to some embodiments, specifies the number of worker subnets 104 that may be determined and / or received algorithmically. As described above with respect to FIG. 1, each of the worker subnets 104 may be assigned one of the subnet addresses from the CIDR slice operation described in FIG. 2. Each worker subnet 104 may include compute nodes 108. The number of worker subnets 306 may be represented using the variable w as an abbreviation in the equations described below. Similar to the number of availability domains 304, the number of worker subnets 306 may be provided per availability domain. In other words, the number of worker subnets 306 may represent the number of worker subnets in each availability domain. For example, the number of worker subnets 306 per availability domain in FIG. 3C is one worker subnet. The number of worker subnets 306 may be received and / or determined by a Terraform script. For example, a user may provide the number of worker subnets 306 per availability domain as an input to the script. Alternatively, the user may select the functions to be performed in the availability domain, and the script may automatically determine which worker subnets should be included in the design, thereby determining the number of worker subnets 306. It may be determined. For example, a user may provide the number of worker subnets 306 per availability domain as an input to the script. Alternatively, the user may select the functions to be performed in the availability domain, and the script may automatically determine which worker subnets should be included in the design, thereby determining the number of worker subnets 306.

[0035] Figure 3D shows a fourth input that may specify the number 308 of load balancer subnets that may be determined and / or received by an algorithm, according to some embodiments. As described above with respect to FIG. 1, each of the load balancer subnets may include a load balancer 112 that manages traffic and routes it to worker subnets. The number of load balancer subnets may be represented as the variable b as an abbreviation in the equations described below. This number may specify the number of load balancer subnets per cluster. Typically, the user need not organize the load balancer subnets into specific availability domains, but rather will simply specify the total number of load balancers required in each cluster. This number may be specified as an input to the script. Alternatively, the number 308 of load balancer subnets may be determined by the script at runtime. For example, the user may specify one or more pre-defined configurations for the service, and the script may derive the number of load balancers required from those configurations.

[0036] Based on these inputs, the following algorithm may be used to determine the total number of subnets to be used for the provisioned service. In one step, the algorithm may calculate the total number of worker subnets to be provisioned. The total number of worker subnets may be calculated based on the total number 302 of clusters, the total number 304 of availability domains per cluster, and the total number 306 of worker subnets per availability domain. For example, the total number of worker subnets may be calculated using the following equation.

[0037]

Number

[0038] In Equation 1, w t represents the total number of worker subnets to be provisioned across the service.

[0039] In another step, the algorithm may calculate the total number of load balancer subnets to be provisioned. The total number of load balancer subnets may be calculated based on the total number of clusters 302 and the total number of load balancers per cluster 308. For example, the total number of load balancer subnets may be calculated using the following formula.

[0040]

Number

[0041] In Equation 2, b t represents the total number of load balancer subnets to be provisioned across the entire service.

[0042] In another step, the algorithm may calculate the total number of subnets to be provisioned in the project. The total number of subnets may be calculated based on the total number of worker subnets and the total number of load balancer subnets. For example, the total number of subnets may be calculated using the following formula.

[0043]

Number

[0044] In Equation 3, s represents the total number of subnets to be allocated across the service. Next, the algorithm may calculate the level at which the top-level network address should be sliced. As described above in connection with FIG. 2, this level may correspond to the base-2 exponential function that determines the number of subnet addresses derived from the top-level network address. In some embodiments, raising the base 2 to the power corresponding to the calculated number of slice levels is the total number of subnets to be allocated in the service (e.g., 2^(number of subnet levels): a power with base 2 and exponent (number of subnet levels)) The above may be the case. To determine the level, the base-2 logarithm of the total number of subnets can be calculated, and then a ceiling operation (e.g., rounding up to the next higher integer) can be applied to the result of the base-2 logarithm to determine the level. For example, a particular embodiment may use the following formula to determine the number of levels.

[0045] [Number]

[0046] Since n ∈ N, the ceiling operation can be applied to Equation 7 to calculate the final level n for the slicing operation.

[0047] [Number]

[0048] Here, using the level number n, a slicing operation may be performed on the top-level network address. In some embodiments, the algorithm may manually calculate the appropriate IP address for each subnet by dividing the network address into 2 n non-overlapping subnets. Depending on the specific IaC tool used to define the architecture, there may be a predefined function that automatically performs this calculation. For example, the Terraform tool provides the cidrsubnet(prefix, newbits, netnum) function that can be used to perform this mathematical calculation for each subnet. In this function, the following inputs may be used. · prefix is a predetermined value derived from the top-level network address. · netnum is a value ∈ [0, 2 - 1] that represents the index of one of the subnet addresses at the nth level from Figure 2. n ​· newbits is a dynamically calculated value of n that represents the minimum value to ensure that there are sufficient CIDR blocks to be assigned to each subnet.

[0049] At this stage, the available subnets can be assigned to each of the resources described in the architecture of Figure 1. In the Terraform tool, this can be achieved using a single count meta-argument for a resource. For example, by inserting the line count = num_worker_subnets after the declaration of that resource in the Terraform code, that resource can be replicated once for each worker subnet. It should be emphasized that IaC languages and tools typically provide a limited looping mechanism that allows only a single level of resources to be replicated.

[0050] However, in practice, it is also useful to generate and / or assign labels based on various combinations of variables. For example, some embodiments may label each subnet according to its combination of cluster and availability domain. When dealing with multiple variables that need to be iterated over to generate labels for each resource, multiple levels of looping may be required. However, as mentioned above, IaC languages do not provide such nested loop control structures as native features of the language. Therefore, in order to assign labels based on multiple variables (e.g., clusters, availability domains, etc.), each such label had to be calculated manually and hard-coded into the script. As mentioned above, hard-coding values into the script limits the reusability and scalability of the design and requires human intervention whenever the design is changed.

[0051] The embodiments described in this specification solve this problem by collecting various label components into a label vector and performing modular arithmetic to generate a Cartesian product. The Cartesian product may then use a single loop feature mimicked by the count meta-argument within the Terraform tool. The loop feature may be used.

[0052] Figure 4 shows how a Cartesian product can be used to form a single list of labels from multiple different variables, according to some embodiments. A first vector containing a list 402 of cluster labels may include clusters referred to as production, development, staging, quality assurance, etc. A second vector containing a list 404 of availability domains may include domains referred to by numerical identifiers (e.g., AD-1, AD-2, etc.). Previously, a script might have statically defined all pairs formed into a single label vector hard-coded by the combination of these two vectors of labels, and then that hard-coded single label vector might have been applied to each of the subnets. However, this limits the reusability of the script. This was necessary because traditional programming languages provide nested loops such that the cluster 402 can be iterated over in an outer loop and the availability domain 404 can be iterated over in an inner loop, but IaC tools typically do not provide such control structures.

[0053] Some embodiments approximate a nested loop control structure by using modular arithmetic as shown in Figure 4. This means that a script can receive any number of clusters, availability domains, or other labels as input and then within the Terraform tool Enables the dynamic generation of a single vector 406 of labels that can be applied using the count meta-arguments of . Using the algorithms described below, the Cartesian product may be generated to include a set of all ordered pairs for each of two input label vectors. It should be emphasized that the two vectors of labels are used here only as an example. Other embodiments may use more than two vector labels, and the Cartesian product may be calculated for each combination of labels to form a single label vector that may be applied without nested looping.

[0054] In FIG. 4, the Cartesian product generated a vector 406 of labels that included each unique pair of labels from cluster 402 and availability domain 404. FIG. 5 shows how the Cartesian product may be calculated using modular arithmetic according to some embodiments. In this example, the Cartesian product is represented as a grid with two-dimensional coordinates. The two-dimensional coordinates represent the indices that would be used in nested loops. For example, the production cluster might loop on index 0, the development cluster on index 1, the staging cluster on index 2, and so on. For the inner loop, each of the availability domains would loop on an index that is incremented from 0 to 2.

[0055] However, instead of using this nested looping process, each pair obtained from the Cartesian product can be indexed using a single index within the resulting vector 406 of labels shown in FIG. 4. The single index is related to the nested indices using modular arithmetic. For example, the index in the outer loop representing the cluster label is related to the index by the following formula.

[0056]

Equation

[0057] The variable a corresponds to the number of values in the input vector, which in this case is 3, and index c corresponds to the cluster index. Similarly, the index of the inner loop representing the availability domain label may be related to the index by the following formula.

[0058]

Number

[0059] index ad corresponds to the availability domain index. Figures 6A - 6E show a detailed example of a Terraform script implementing the above - mentioned algorithm This flowchart and pseudocode are provided as examples and are not meant to be limiting. The algorithms described above may be implemented using any IaC language using the present disclosure.

[0060] Figure 6A shows the first part of a flowchart of a method for provisioning a service in a cloud infrastructure, along with accompanying pseudocode, according to some embodiments. The method may include receiving state variables and collecting data from the cloud infrastructure (602). As described above, some embodiments may receive each of the variables defining the resources to be provisioned as inputs to the script. In this example, this may include the number of availability domains, the number of worker subnets in each domain, the number of load - balancer subnets per cluster, etc. In this example, it may be assumed that the number of worker subnets per availability domain is 1 and the number of load - balancer subnets per cluster is 2. The pseudocode 601 for receiving these values may include assigning each of these values to a variable. The pseudocode 601 may also include the FastConnect user Utility is also used, which is an alternative network connection to using the public Internet to connect to cloud infrastructure and other online services.

[0061] Pseudo-code 601 also includes an implementation example of the algorithm described above to define the number of CIDR levels. For example, the above Equation 8 is implemented in the last line of Pseudo-code 601:

[0062]

Number

[0063] This line of pseudo-code calculates the total number of subnets required for the load balancer and worker nodes. This total is provided as input to the base-2 logarithmic function, and the result is provided to the ceiling function to determine the minimum number of CIDR levels for the service.

[0064] The method may also include provisioning a gateway for the service (604). Pseudo-code 603 shows an example of how to define a service gateway, an Internet gateway, a NAT gateway, etc. As described above, the syntax of Pseudo-code 603 may be specific to the Terraform tool, and thus, it may be changed when working with other tools.

[0065] Figure 6B shows the continuation of a flowchart of a method for provisioning services in a cloud infrastructure according to some embodiments. The method may further include configuring a routing table and security rules for the service (606). Pseudocode 605 shows an example for declaring a default routing table and a load balancer security list. Note that some additional security and / or routing resources may be declared in a real-world project. The example provided in Pseudocode 605 is merely representative and may be used as a template for generating other security rules, security lists, and routing tables for different aspects of the service.

[0066] Figure 6C shows the continuation of a flowchart of a method for provisioning services in a cloud infrastructure according to some embodiments. The method may further include determining whether all worker subnets have been provisioned (608). If there are remaining worker subnets to be provisioned, the method may include calculating the subnet addresses using the CIDR slicing algorithm described above (610). Each of the subnets may then be provisioned with the corresponding CIDR block (612). This cycle may be repeated for each of the worker subnets.

[0067] As is apparent in Pseudocode 614, the loop construct shown in the flowchart cannot be represented in Terraform pseudocode. Instead, this code provides a meta-argument that replicates the resource a specified number of times. For example, the first line of the pseudocode after the resource declaration includes: provides a meta-argument that replicates the resource a specified number of times. For example, the first line of the pseudocode after the resource declaration includes:

[0068]

Number

[0069] By assigning the number of worker subnets to a counter variable, the pseudo-code 614 can instruct the Terraform tool to generate oci_core_subnet resources for each worker subnet. Next, the pseudo-code 614 uses the above cidrsubnet() with the count.index variable that is used to index each of the subnets at the specified level calculated above.

[0070] The pseudo-code 614 also implements the nested loop algorithm described above by calculating the Cartesian product between the cluster name and the availability domain name. These are stored in the availability_domain array indexed by the count.index variable. For example, the following command generates an array of labels for each of the subnets:

[0071]

Number

[0072] Instead of executing the traditional nested loop, this Cartesian product generates a list of labels for the subnets. Finally, the labels can be assigned to the corresponding subnets using the modular arithmetic shown in pseudo-code 614 and described in detail above.

[0073] The process shown in Figure 6C assigns each of the worker subnets in the project It may be executed for application. Although not specifically shown, the process shown in FIG. 6C may be repeated for each of the load balancer subnets. For example, following a similar procedure, the top-level network address may be sliced based on the levels calculated above, and a list of labels may be generated using the Cartesian product. However, the netnum parameter for the cidrsubnet() function may be shifted so that the load balancer subnets are assigned to CIDR blocks after the worker subnets.

[0074] FIG. 6D shows a continuation of a flowchart of a method for provisioning services in a cloud infrastructure according to some embodiments. The method may further include determining (616) whether a sufficient number of clusters have been provisioned based on the number of clusters described above. If a cluster remains to be provisioned, the method may determine (618) which load balancer subnets are required and provision the correct load balancer for the cluster (620). As described above, this loop procedure may be modeled in pseudocode 622 to dynamically generate a specified number of clusters using a count variable. The second part of pseudocode 622 may assign load balancer subnets to each of the clusters.

[0075] FIG. 6E completes a flowchart of a method for provisioning services in a cloud infrastructure according to some embodiments. The method may also include determining (624) whether a sufficient node pool has been provisioned. As long as additional node pools need to be provisioned, the method is ChunkList The method may include dividing the subnet into node pools for allocation (626) using a family map operation. In pseudocode 630, a ChunkList function from the Terraform tool may be used to divide the single list into fixed-sized chunks, thus returning a list of lists. The method may then include provisioning the node pools with corresponding worker subnets (628). In pseudocode 630, each node pool has a specified quantity of nodes per subnet that is described in a family map.

[0076] The above flowchart and explanations are specific to the Terraform tool, but CIDR subnetting is not supported. An algorithm for simulating nested loops to generate and assign subnets and apply labels to the subnets may be implemented in any IaC language. FIG. 7 illustrates a flowchart of a method for provisioning a variable number of subnets with corresponding labels at runtime, which may be implemented in any IaC language, according to some embodiments. The method may include determining a number of clusters to be provisioned as part of a service in a cloud infrastructure (702). The method may also include determining a number of availability domains in each of the clusters (704), determining a number of compute nodes in each of the availability domains (706), and / or determining a number of load balancers in each of the clusters (708). In each of these steps, these values ​​may be determined by being received as input to a script, by being stored as constants in the script, and / or by being determined by a calculation or function in the script. For example, each value may be derived or looked up based on a predefined configuration selected by a user. In some embodiments, these values ​​may be determined as described above in connection with FIGS. 3A-3B or as shown in FIG. 6A.

[0077] The method may also include calculating (710) the number of subnetwork levels based on the number of clusters, the number of availability domains, the number of compute nodes, and the number of load balancers. The compute nodes may correspond to the worker subnets described above. Each of the load balancers may also be described as a load balancer subnet. The number of subnetwork levels may be calculated as described above in connection with Equation 2 - Equation 8 and in connection with FIG. 6C. The method may further include generating (712) a plurality of subnets based on the number of subnetwork levels. This step may be performed using the cidrsubnet function described above. Alternatively or additionally, this step may be performed by mathematically subdividing the top-level network address based on the subdivision of 2 n . The method may also include allocating (714) the plurality of subnets to compute nodes and load balancers within the cloud infrastructure. As described above, each of the worker subnets and load balancer subnets may be assigned a label based on the Cartesian product and modular arithmetic algorithms described above.

[0078] It should be understood that the specific steps shown in FIG. 7 provide a specific way of provisioning variable subnets, along with corresponding labels, at runtime, according to various embodiments. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Further, the individual steps shown in FIG. 7 may include multiple sub-steps that may be performed in various sequences appropriate for the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0079] Each of the methods described herein may be implemented by a computer system. Each step of these methods may be automatically executed by a computer system and / or input / output involving the user may be provided. For example, the user may provide input for each step of a method, and each of these inputs may be in response to a specific output that requires such input, and that output is generated by the computer system. Each input may be received in response to the corresponding required output. Further, the input may be received from the user, as a data stream from another computer system, retrieved from a memory location, retrieved via a network, or requested from a web service. Similarly, the output may be provided to the user, as a data stream to another computer system, stored in a memory location, transmitted via a network, or provided to a web service. In short, each step of the methods described herein may be performed by a computer system, with or without user involvement, and may include any number of inputs, outputs, and / or requests to and from the computer system. Steps without user involvement may be said to be automatically executed by the computer system without human intervention. Thus, in light of the present disclosure, it will be understood that each step of each method described herein may be modified to include input from and output to the user, or may be automatically performed by the computer system without human intervention if any decisions are made by the processor. Further, some embodiments of each of the methods described herein may be implemented as a set of instructions stored on a tangible, non-transitory storage medium to form a tangible software product.

[0080] FIG. 8 shows a simplified diagram of a distributed system 800 for implementing one of the embodiments. In the illustrated embodiment, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808, which are configured to execute and operate client applications such as web browsers, proprietary clients (e.g., Oracle Forms), etc. via one or more networks 810. Server 812 may be communicatively coupled to remote client computing devices 802, 804, 806, and 808 via network 810.

[0081] In various embodiments, server 812 may be adapted to execute one or more services or software applications provided by one or more of the components of the system. In some embodiments, these services may be provided to users of client computing devices 802, 804, 806, and / or 808 as web-based services or cloud services, or under a software-as-a-service (SaaS) model. The users operating client computing devices 802, 804, 806, and / or 808 may then interact with server 812 using one or more client applications to utilize the services provided by these components.

[0082] In the configuration shown in the figure, software components 818, 820, and 822 of system 800 are shown as being implemented on server 812. In other embodiments, one or more of the components of system 800 and / or the services provided by these components may be implemented by one or more of client computing devices 802, 804, 806, and / or 808. A user operating a client computing device may then use one or more client applications to utilize the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that various different system configurations are possible that may differ from the distributed system 800. The embodiment shown in the figure is, therefore, an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0083] Client computing devices 802, 804, 806, and / or 808 may be portable handheld devices (e.g., iPhone (registered trademark), cellular phone, iPad (registered trademark), computing tablet, personal digital assistant (PDA)) or may be wearable devices (e.g., Google Glass (registered trademark) head-mounted display), software such as Microsoft Windows Mobile (registered trademark), and / or various operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS Runs a byte operating system and is an effective communication protocol such as the Internet, email, Short Message Service (SMS), Blackberry (registered trademark), or others. The client computing device can be a general-purpose personal computer, and as an example, various versions of Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux (registered trademark) operating systems can be run on personal computers and / or laptop computers. The client computing device includes, but is not limited to, various GNU / Linux (registered trademark) operating systems such as Google Chrome OS, and can be a workstation computer running any of various commercially available UNIX (registered trademark) or UNIX-like operating systems. Alternatively, or in addition, client computing devices 802, 804, 806, and 808 can be any other electronic device capable of communicating via network 810, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device. including personal computers and / or laptop computers running various versions thereof, and can include various commercially available UNIX (registered trademark) or UNIX-like operating systems. The client computing device can be a workstation computer running any of various commercially available UNIX (registered trademark) or UNIX-like operating systems. Alternatively, or in addition, client computing devices 802, 804, 806, and 808 can be any other electronic device capable of communicating via network 810, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device.

[0084] The exemplary distributed system 800 is shown with four client computing devices, but any number of client computing devices can be supported. Other devices, such as devices with sensors, can interact with server 812.

[0085] The network 810 within the distributed system 800 is TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Exchange), AppleTalk, and the like, including but not limited to various Any type of network well known to those skilled in the art that can support data communication using any of a variety of commercially available protocols, such as, by way of example only, Network 810 may be a local area network (LAN) based on Ethernet (registered trademark), Token Ring, or the like. Network 810 may be a wide area network and the Internet. It may be a virtual private network (VPN), intranet, extranet, public switched telephone network (PSTN), infrared network, wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, Bluetooth (registered trademark), and / or any other wireless protocol), including but not limited to virtual networks; and / or any combination of these and / or other networks.

[0086] Server 812 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (Personal Computer) servers, UNIX (registered trademark) servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. In various embodiments, Server 812 may be adapted to execute one or more of the services or software applications described in the foregoing disclosure. For example, Server 812 may correspond to a server for executing the processes described above in accordance with an embodiment of the present disclosure.

[0087] Server 812 may execute an operating system including any of the above, and any server operating system available on the market. Server 812 may also execute any of a variety of other server applications and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA (registered trademark) server, a database server, etc. Exemplary database servers include, but are not limited to, those available on the market from Oracle, Microsoft, Sybase, IBM (registered trademark) (International Business Machines), etc.

[0088] In some embodiments, server 812 may include one or more applications for analyzing and collating data feeds and / or event updates received from users of client computing devices 802, 804, 806, and 808. As an example, the data feeds and / or event updates may include real-time events related to sensor data applications, financial stock market dashboards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc., received from one or more third-party information sources and continuous data streams, and may include Twitter (registered trademark) feeds, Facebook (registered trademark) updates or real-time updates, but are not limited thereto. Server 812 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 802, 804, 806, and 808.

[0089] The distributed system 800 may also include one or more databases 814 and 816. The databases 814 and 816 may be located in various locations. By way of example, one or more of the databases 814 and 816 may reside on a non-transitory storage medium that is local to (and / or resident on) the server 812. Alternatively, the databases 814 and 816 may be remote from the server 812 and communicate with the server 812 via a network-based connection or a dedicated connection. In a set of embodiments, the databases 814 and 816 may reside within a storage area network (SAN). Similarly, any necessary files for performing functions attributable to the server 812 may be stored locally on the server 812 and / or remotely, as appropriate. In a set of embodiments, the databases 814 and 816 may include a relational database, such as a database provided by Oracle, that is adapted to store, update, and retrieve data in response to SQL-formatted commands.

[0090] FIG. 9 is a simplified block diagram of one or more components of a system environment 900 that may provide, as cloud services, services provided by one or more components of an embodiment system according to an embodiment of the present disclosure. In the illustrated embodiment, the system environment 900 includes one or more client computing devices 904, 906, and 908 that may be used by a user to interact with a cloud infrastructure system 902 that provides cloud services. The client computing devices may be configured to operate a client application, such as a web browser, a client application under intellectual property rights (e.g., Oracle Forms), or some other application, that may be used by a user of the client computing device to interact with the cloud infrastructure system 902 to use services provided by the cloud infrastructure system 902.

[0091] It should be understood that the illustrated cloud infrastructure system 902 may have components other than those illustrated. Further, the embodiments shown in the figures are only examples of cloud infrastructure systems that may incorporate embodiments of the present invention. In some other embodiments, the cloud infrastructure system 902 may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or arrangements of components.

[0092] Client computing devices 904, 906, and 908 may be devices similar to those described above for 802, 804, 806, and 808.

[0093] The exemplary system environment 900 is shown with three client computing devices, but any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the cloud infrastructure system 902.

[0094] Network 910 may facilitate data communication and exchange between clients 904, 906, and 908 and the cloud infrastructure system 902. Each network may be any type of network well known to those skilled in the art that can support data communication using any of a variety of commercially available protocols, including those described above for network 810.

[0095] The cloud infrastructure system 902 may include one or more computers and / or servers that may include those described above for server 812.

[0096] In certain embodiments, the services provided by the cloud infrastructure system S may include hosting services that are made available to users of an on-demand cloud infrastructure system, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, etc. Services provided by the cloud infrastructure system can scale dynamically to meet the needs of its users. A particular instantiation of a service provided by the cloud infrastructure system is referred to herein as a "service instance". Generally, any service made available to a user via a communication network such as the Internet from a cloud service provider's system is referred to as a "cloud service". Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. For example, the cloud service provider's system may host an application, and the user may order and use that application on demand via a communication network such as the Internet.

[0097] In some examples, services within a computer network cloud infrastructure may include protected computer network access to storage, hosted databases, hosted web servers, software applications, or other services provided to users by a cloud vendor or known in the art. For example, a service can include password-protected access to remote storage on the cloud via the Internet. As another example, a service can include a web service-based hosted relational database and a scripting language middleware engine for private use by networked developers. As another example, a service can include access to an email software application hosted on a cloud vendor's website.

[0098] In one embodiment, the cloud infrastructure system 902 may include a set of application, middleware, and database service offerings that are self-service, subscription-based, elastically scalable, reliable, highly available, and delivered to customers in a secure fashion. An example of such a cloud infrastructure system is the Oracle Public Cloud provided by the present assignee.

[0099] In various embodiments, the cloud infrastructure system 902 may be adapted to automatically provision, manage, and track a customer's subscription to services provided by the cloud infrastructure system 902. The cloud infrastructure system 902 may provide cloud services via different deployment models. For example, the services may be provided under a public cloud model where the cloud infrastructure system 902 is owned by an organization that sells cloud services (e.g., owned by Oracle) and the services are made available to the general public or different industry enterprises. As another example, the services may be provided under a private cloud model where the cloud infrastructure system 902 operates only for a single organization and provides the services to one or more entities within that organization. The cloud services may also be provided under a community cloud model where the cloud infrastructure system 902 and the services provided by the cloud infrastructure system 902 are shared by several organizations within a related community. The cloud services may also be provided under a hybrid cloud model that is a combination of two or more different models.

[0100] In some embodiments, the services provided by the cloud infrastructure system 902 may include one or more services provided under other categories of services, including software as a service (SaaS) category, platform as a service (PaaS) category, infrastructure as a service (IaaS) category, or hybrid services. A customer may order one or more services provided by the cloud infrastructure system 902 via a subscription order. The cloud infrastructure system 902 then performs processing to provide the services in the customer's subscription order.

[0101] In some embodiments, the services provided by the cloud infrastructure system 902 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, the application services may be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform may be configured to provide cloud services corresponding to the SaaS category. For example, the SaaS platform may provide the ability to build and deliver a suite of on-demand applications on an integrated development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure for providing the SaaS services. By utilizing the services provided by the SaaS platform, a customer can utilize applications running on the cloud infrastructure system. The customer can obtain the application services without having to purchase separate licenses and support. Various different SaaS services may be provided. Examples include, but are not limited to, services that provide solutions for sales performance management, enterprise integration, and business agility for large organizations.

[0102] In some embodiments, the platform service may be provided by a cloud infrastructure system via a PaaS platform. The PaaS platform may be configured to provide cloud services corresponding to the PaaS category. Examples of platform services may include, but are not limited to, services that enable an organization (such as Oracle) to integrate existing applications on a shared common architecture, and the ability to build new applications that utilize shared services provided by the platform. The SaaS platform may manage and control the underlying software and infrastructure for providing SaaS services. Customers can obtain PaaS services provided by the cloud infrastructure system without the need to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), etc.

[0103] By using the services provided by the PaaS platform, customers can adopt programming languages and tools supported by the cloud infrastructure system and can also control the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware services), and Java cloud services. In one embodiment, the database cloud service may support a shared service deployment model that enables organizations to pool database resources and provide databases as services to customers in the form of a database cloud. The middleware cloud service may provide a platform for customers to develop and deploy various business applications, and the Java cloud service may provide a platform for customers to deploy Java applications in the cloud infrastructure system.

[0104] Various different infrastructure services may be provided by the IaaS platform in the cloud infrastructure system. The infrastructure services facilitate the management and control of underlying computing resources such as storage, network, and other basic computing resources for customers who utilize the services provided by the SaaS platform and the PaaS platform.

[0105] In one embodiment, the cloud infrastructure system 902 may also include infrastructure resources 930 for providing resources used to provide various services to customers of the cloud infrastructure system. In one embodiment, the infrastructure resources 930 may include a pre-integrated and optimized combination of hardware such as servers, storage, and networking resources to execute services provided by the PaaS platform and the SaaS platform.

[0106] In some embodiments, the resources in the cloud infrastructure system 902 may be shared by multiple users and dynamically reallocated as needed. Additionally, the resources may be allocated to users at different times. For example, the cloud infrastructure system 930 may enable a first set of users in a first time period to utilize the resources of the cloud infrastructure system for a specified number of hours and then reallocate the same resources to another set of users in a different time period, thereby maximizing resource utilization.

[0107] In one embodiment, some internal shared services 932 may be provided that are shared by different components or modules of the cloud infrastructure system 902 and by the services provided by the cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0108] In certain embodiments, the cloud infrastructure system 902 may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one embodiment, the cloud management functionality may include, among other things, the ability to provision, manage, and track customer subscriptions received by the cloud infrastructure system 902.

[0109] In one embodiment, as shown in the figure, the cloud management functionality may be provided by one or more modules such as an order management module 920, an order orchestration module 922, an order provisioning module 924, an order management and monitoring module 926, and an identity management module 928. These modules may include or be provided using one or more computers and / or servers that may be general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable configuration and / or combination thereof.

[0110] In exemplary operation 934, a customer using a client device such as client devices 904, 906, or 908 may interact with the cloud infrastructure system 902 by requesting one or more services provided by the cloud infrastructure system 902 and placing an order for a subscription to one or more services provided by the cloud infrastructure system 902. In certain embodiments, the customer may access a cloud user interface (UI), cloud UI 912, cloud UI 914, and / or cloud UI 916 and place a subscription order through these UIs. Order information received by the cloud infrastructure system 902 in response to the customer placing an order may include information identifying the customer and the one or more services provided by the cloud infrastructure system 902 that the customer wishes to subscribe to.

[0111] After an order is placed by a customer, order information is received via the cloud UIs 912, 914, and / or 916.

[0112] In operation 936, the order is stored in the order database 918. The order database 918 can be one of several databases that are operated by the cloud infrastructure system 918 and are operated in relation to other system elements.

[0113] In operation 938, the order information is transferred to the order management module 920. In some examples, the order management module 920 may be configured to perform billing and accounting functions related to the order, such as verifying the order and, if verified, booking the order.

[0114] In operation 940, information about the order is communicated to the order orchestration module 922. The order orchestration module 922 may use the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some examples, the order orchestration module 922 may use the services of the order provisioning module 924 to orchestrate the provisioning of resources to support the subscribed services.

[0115] In certain embodiments, the order orchestration module 922 enables the management of the business processes associated with each order and applies business logic to determine whether an order should proceed to provisioning. In operation 942, upon receiving an order for a new subscription, the order orchestration module 922 allocates resources and sends a request to the order provisioning module 924 to configure the resources required to fulfill the subscription order. The order provisioning module 924 enables the allocation of resources for the services ordered by the customer. The order provisioning module 924 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 900 and the physical implementation layer used to provision the resources for providing the requested services. Thus, the order orchestration module 922 may be isolated from the implementation details, such as whether the services and resources are actually provisioned on-the-fly or pre-provisioned and only allocated / assigned upon request.

[0116] In operation 944, upon the services and resources being provisioned, a notification of the provided services may be sent to the customer on the client devices 904, 906, and / or 908 by the order provisioning module 924 of the cloud infrastructure system 902.

[0117] In operation 946, the customer's subscription order may be managed and tracked by the order management and monitoring module 926. In some cases, the order management and monitoring module 926 may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system up-time and system down-time.

[0118] In one embodiment, the cloud infrastructure system 900 may include an identity management module 928. The identity management module 928 may be configured to provide identity services such as access management and authorization services in the cloud infrastructure system 900. In some embodiments, the identity management module 928 may control information regarding customers who desire to utilize services provided by the cloud infrastructure system 902. Such information may include information for authenticating the identities of such customers and information describing what actions they are permitted to perform on various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). The identity management module 928 may also include management of descriptive information about each customer, as well as how and who can access and modify that descriptive information.

[0119] FIG. 10 shows an exemplary computer system 1000 in which various embodiments of the present invention may be implemented. The system 1000 may be used to implement any of the computer systems described above. As shown in the figure, the computer system 1000 includes a processing unit 1004 that communicates with several peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and a system memory 1010.

[0120] The bus subsystem 1002 provides a mechanism for enabling the various components and subsystems of the computer system 1000 to communicate with each other as intended. Although the bus subsystem 1002 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1002 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus, using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Extended ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0121] The processing unit 1004 can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller) and controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core processors or multi-core processors. In certain embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, each of which includes a single-core processor or multi-core processor. In other embodiments, the processing unit 1004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0122] In various embodiments, the processing unit 1004 can execute various programs in response to program code and can maintain multiple simultaneously executed programs or processes. At any given time, some or all of the program code to be executed can reside in the processor 1004 and / or the storage subsystem 1018. Through suitable programming, the processor 1004 can provide the various functionalities described above. The computer system 1000 may further include a processing acceleration unit 1006 that can include a digital signal processor (DSP), a special-purpose processor, and the like.

[0123] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a mouse or other pointing device such as a trackball, a touchpad or touch screen incorporated in a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may enable a user to control and interact with input devices such as a Microsoft Xbox (registered trademark) 360 game controller through a natural user interface using gestures and spoken commands, and may include motion sensing and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor. User interface input devices may detect eye movements from the user (e.g., "blinks" while taking a photo and / or making a menu selection) and input eye gestures to a device (e.g., Google Eye gesture recognition devices such as a Google Glass (registered trademark) blink detector that converts to an input to Glass (registered trademark) may also be included. Additionally, the user interface input device may include an audio recognition sensing device that enables the user to interact with an audio recognition system (e.g., Siri (registered trademark) navigator) via voice commands.

[0124] The user interface input device may include, but is not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, the user interface input device may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound examination devices. The user interface input device may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0125] The user interface output device may include a non-visual display such as a display subsystem, an indicator light, or an audio output device. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), a liquid crystal display (LCD), or a plasma display, a projection device, a touch screen, etc. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1000 to the user or another computer. For example, the user interface output device may include, but is not limited to, various display devices for visually conveying text, graphics, and audio / video information, such as a monitor, a printer, a speaker, headphones, an automotive navigation system, a plotter, an audio output device, and a modem.

[0126] The computer system 1000 may include a storage subsystem 1018 that includes software elements shown as being currently located within the system memory 1010. The system memory 1010 may store program instructions loadable and executable on the processing unit 1004, as well as data generated during the execution of these programs.

[0127] Depending on the configuration and type of the computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to the processing unit 1004 and / or are currently being operated on and executed by the processing unit 1004. In some implementations, the system memory 1010 may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS) that includes basic routines useful for transferring information between elements within the computer system 1000, such as during startup, may typically be stored in ROM. By way of example and not limitation, the system memory 1010 also shows an application program 1012, program data 1014, and an operating system 1016 that may include, for example, client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc. By way of example, the operating system 1016 may include various versions of the Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux (registered trademark) operating systems, various commercially available UNIX (registered trademark) or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome (registered trademark) OS, etc.), and / or mobile operating systems such as iOS, Windows (registered trademark) Phone, Android (registered trademark) OS, BlackBerry (registered trademark) 10 OS, and Palm (registered trademark) OS.

[0128] The memory subsystem 1018 may also provide a tangible computer-readable storage medium for storing the basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the above-described functionality when executed by a processor may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with the present invention.

[0129] The storage subsystem 1000 may also include a computer-readable storage medium reader 1020 that may be further connected to a computer-readable storage medium 1022. Together with the system memory 1010 and, optionally, in combination with the system memory 1010, the computer-readable storage medium 1022 may comprehensively represent a storage medium added to a remote, local, fixed, and / or removable storage device for temporarily and / or more permanently accommodating, storing, transmitting, and retrieving computer-readable information.

[0130] The computer-readable storage medium 1022 that includes code or a portion of code may also include any suitable medium known in or used in the art, including, but not limited to, volatile and non-volatile, removable and non-removable media, such as storage media and communication media, implemented by any method or technology for the storage and / or transmission of information. It can be. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or other tangible computer-readable media. This may also include non-tangible computer-readable media such as data signals, data transmissions, or any other media that can be used to transmit the desired information and can be accessed by computing system 1000.

[0131] As an example, computer-readable storage medium 1022 may include a hard disk drive that reads and writes to non-removable non-volatile magnetic media, a magnetic disk drive that reads and writes to removable non-volatile magnetic disks, an optical disk drive that reads and writes to removable non-volatile optical disks such as CD ROM, DVD and Blu-Ray (registered trademark ) disks, or other optical media. Computer-readable storage medium 1022 may include, but is not limited to, Zip (registered trademark) drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tapes, etc. Computer-readable storage medium 1022 may also include solid state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid state ROMs, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and the computer-readable media associated therewith may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data to computing system 1000.

[0132] The communication subsystem 1024 provides an interface to other computer systems and networks. The communication subsystem 1024 serves as an interface for the transfer of data between other systems and the computer system 1000. For example, the communication subsystem 1024 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1024 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G or EDGE (Enhanced Data rates for GSM Evolution), WiFi (IEEE802.11 family of standards, or other mobile communication technologies, or any combination thereof), a Global Positioning System (GPS) receiver component, and / or other components. In some embodiments, the communication subsystem 1024 can provide wired network connectivity (e.g., Ethernet (registered trademark)) in addition to, or instead of, a wireless interface.

[0133] In some embodiments, the communication subsystem 1024 may also receive input communications on behalf of one or more users who may use the computer system 1000, in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc.

[0134] As an example, the communication subsystem 1024 may receive data feeds 10 in real time from users of social networks and / or other communication services, such as Twitter (registered trademark) feeds, Facebook( registered trademark) updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources. It may be configured to receive 26.

[0135] In addition, the communication subsystem 1024 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1028 and / or event updates 1030 of real-time events that may be essentially continuous or infinite without an explicit end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock market dashboards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0136] The communication subsystem 1024 may also be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. to one or more databases that communicate with one or more streaming data source computers coupled to the computer system 1000.

[0137] The computer system 1000 can be one of various types, including a handheld portable device (e.g., an iPhone (registered trademark) mobile phone, an iPad (registered trademark) computing tablet, a PDA), a wearable device (e.g., a Google Glass (registered trademark) head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0138] Due to the constantly changing nature of computers and networks, the description of the computer system 1000 shown in the figures is intended merely as a specific example. Many other configurations are possible that have more or fewer components than the systems depicted in the figures. For example, customized hardware may also be used and / or certain elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other aspects and / or methods for implementing various embodiments.

[0139] In the foregoing description, for purposes of explanation, numerous specific details were set forth in order to provide a thorough understanding of the various embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.

[0140] The foregoing description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the foregoing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and configuration of the elements without departing from the spirit and scope of the invention as set forth in the claims.

[0141] In the above description, specific details are given to provide a complete understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may sometimes be shown as components in block diagram form in order not to obscure the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may sometimes be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0142] Also, note that individual embodiments may sometimes be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations can be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0143] The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and various other media that can store, contain, or carry instructions and / or data. A code segment or machine-executable instruction may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, transferred, or transmitted via any suitable means including, but not limited to, memory sharing, message passing, token passing, network transmission, etc.

[0144] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments performing the necessary tasks may be stored on a machine-readable medium. The necessary tasks may be performed by a processor.

[0145] In the foregoing specification, aspects of the invention have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. The various features and aspects of the above invention may be used individually or in combination. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0146] Furthermore, for purposes of illustration, the method has been described in a particular order. It should be understood that in alternative embodiments, the method may be performed in an order different from that described. Also, the above method may be performed by hardware components or may be embodied as a sequence of machine-executable instructions that, when used, cause a machine such as a general or special purpose processor or logic circuit programmed with such instructions to perform the above method. It should also be understood that these machine-executable instructions may be stored on one or more machine-readable media such as a CD-ROM or other type of optical disk, a floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the method may be performed by a combination of hardware and software.

Claims

1. 1. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: determining a number of clusters to be provisioned as part of a service in a cloud infrastructure; determining a number of availability domains for each of the clusters to be provisioned in the cloud infrastructure; determining a number of compute nodes in each of the availability domains to be provisioned in the cloud infrastructure; determining a number of load balancers for each of the clusters to be provisioned in the cloud infrastructure; Calculating a number of subnet levels based on a number of the clusters, the number of the availability domains, the number of the compute nodes, and the number of the load balancers; generating a number of subnets based on the number of subnet levels; and allocating the plurality of subnets to the compute nodes and the load balancers in the cloud infrastructure.

2. The computer-readable medium of claim 1 , wherein the operations further include causing each of the compute nodes and each of the load balancers to be provisioned with the multiple subnets in the cloud infrastructure.

3. The operation further comprises: receiving a change to the number of availability domains, the number of clusters, the number of compute nodes, or the number of load balancers; recalculating the number of subnet levels based on the changes; and and regenerating the plurality of subnets based on the number of subnet levels.

4. The operation further comprises: determining a second number of clusters, a second number of availability domains, a second number of compute nodes, and a second number of load balancers to be provisioned in the cloud infrastructure as part of a second service, the second service being different from the service, the operations further comprising: calculating a second number of subnet levels based on the second number of clusters, the second number of availability domains, the second number of compute nodes, and the second number of load balancers, the second number of subnet levels being different from the number of subnet levels, the operations further comprising: The computer-readable medium of claim 1 , further comprising generating a second plurality of subnets based on the second number of subnet levels.

5. 2. The method of claim 1, wherein the number of the plurality of subnets is equal to base 2^(number of subnet levels). A computer readable medium as described herein.

6. Determining the number of availability domains includes: receiving a selection of a configuration from a plurality of predefined configurations for the service; and determining a service level agreement (SLA) for the configuration, the SLA defining the number of availability domains.

7. Determining the number of availability domains includes:

2. The computer-readable medium of claim 1, wherein the instructions, when executed by the one or more processors, comprise receiving the number of availability domains as a runtime input.

8. 2. The computer-readable medium of claim 1, wherein an availability domain in the availability domain includes at least one of the compute nodes and at least one of the load balancers.

9. generating the plurality of subnets based on the number of subnet levels, 10. The computer-readable medium of claim 1, comprising performing a function to perform Classless Inter-Domain Routing (CIDR) slicing of a top-level network address.

10. 10. The computer-readable medium of claim 9, wherein the function accepts the number of subnet levels as a parameter such that the number of subnet levels can be calculated at run time.

11. The computer-readable medium of claim 1 , wherein the operations further comprise assigning a plurality of labels to each of the plurality of subnets.

12. The computer-readable medium of claim 11 , wherein the plurality of labels is based on a combination of the availability domain and the cluster.

13. 13. The computer-readable medium of claim 12, wherein the operations further comprise generating the combinations of the availability domains and the clusters by computing a Cartesian product.

14. 13. The computer-readable medium of claim 12, wherein computing the Cartesian product comprises generating a vector cross product between the availability domain and the cluster.

15. 15. The computer-readable medium of claim 14, wherein an index at the plurality of labels is related to an index at the cluster by dividing the index at the plurality of labels by a number of the clusters.

16. 15. The computer-readable medium of claim 14, wherein the indexes at the plurality of labels are related to indexes at the availability domains by performing a modular division of the indexes at the plurality of labels by the number of availability domains.

17. 13. The computer-readable medium of claim 12, wherein the plurality of labels includes an index that is modified by a counter that is incremented each time a compute node or load balancer is declared.

18. The computer-readable medium of claim 12 , wherein the plurality of labels is generated without the use of nested loop structures.

19. 1. A system comprising: one or more processors; and one or more memory devices containing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: determining a number of clusters to be provisioned as part of a service in a cloud infrastructure; determining a number of availability domains for each of the clusters to be provisioned in the cloud infrastructure; determining a number of compute nodes in each of the availability domains to be provisioned in the cloud infrastructure; determining a number of load balancers for each of the clusters to be provisioned in the cloud infrastructure; Calculating a number of subnet levels based on a number of the clusters, the number of the availability domains, the number of the compute nodes, and the number of the load balancers; generating a number of subnets based on the number of subnet levels; and allocating the plurality of subnets to the compute nodes and the load balancers in the cloud infrastructure.

20. 1. A method for executing a provisioning script with runtime adjustment resource declarations, comprising: determining a number of clusters to be provisioned as part of a service in a cloud infrastructure; determining a number of availability domains for each of the clusters to be provisioned in the cloud infrastructure; determining a number of compute nodes in each of the availability domains to be provisioned in the cloud infrastructure; determining a number of load balancers for each of the clusters to be provisioned in the cloud infrastructure; Calculating a number of subnet levels based on a number of the clusters, the number of the availability domains, the number of the compute nodes, and the number of the load balancers; generating a number of subnets based on the number of subnet levels; and allocating the plurality of subnets to the compute nodes and the load balancers in the cloud infrastructure.

Images