Multi-cluster Warehouse

Abstract

To provide a method for a multi-cluster warehouse which includes allocating a plurality of compute clusters as a part of a virtual warehouse.SOLUTION: Compute clusters are used for executing queries by accessing one or more databases in one or more cloud storage resources. A method includes providing each of the plurality of compute clusters with the queries to a virtual warehouse. Each of the plurality of compute clusters for the virtual warehouse receives the plurality of queries so that a computing load is spread across the different clusters. The method also includes, as needed, dynamically adding the compute clusters to the virtual warehouse and removing the compute clusters from the virtual warehouse, based on a workload of the plurality of compute clusters.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Application No. 62 / 328,943, entitled "Multi-Cluster Warehouse," filed April 28, 2016, which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to systems, methods, and devices for multi-cluster warehouses. [Background technology]

[0003] Databases are widely used in computing applications to store and access data. A database may include one or more tables that contain or reference data that can be retrieved, modified, or deleted using queries. A database can store small or very large data sets within one or more tables. This data can be accessed by various users of an organization or used to provide to public users, such as through a website or application program interface (API). Both computing and storage resources and their underlying architecture can play a large role in achieving desired database performance.

[0004] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following drawings, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating a processing platform according to an exemplary embodiment of the systems and methods described herein. [Figure 2] FIG. 2 is a block diagram illustrating components of a resource manager according to one embodiment. [Figure 3]FIG. 1 is a block diagram illustrating scheduling for a multi-cluster warehouse, according to one embodiment. [Figure 4] FIG. 1 is a block diagram illustrating multiple warehouses that may be provided on a single execution platform, according to one embodiment. [Figure 5] FIG. 1 is a block diagram illustrating a system having multiple distributed virtual warehouses, according to one embodiment. [Figure 6] FIG. 1 is a schematic flow chart diagram illustrating a method for a multi-cluster warehouse, according to one embodiment. [Figure 7] FIG. 1 is a schematic flow chart diagram illustrating a method for dynamically adding computer clusters in a multi-cluster warehouse, according to one embodiment. [Figure 8] FIG. 1 is a schematic flow chart diagram illustrating a method for dynamically removing a computer cluster in a multi-cluster warehouse, according to one embodiment. [Figure 9] FIG. 1 is a schematic flow chart diagram illustrating a method for a multi-cluster warehouse, according to one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an example computing device consistent with at least one embodiment of the processes and systems disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0006] The present disclosure is directed to systems, methods, and devices for providing and managing a multi-cluster warehouse. A warehouse is a number of servers connected or cooperating in handling analytical queries. In some warehouses, computational and storage resources are interconnected and allocated. In at least some embodiments disclosed herein, computational resources are independently allocated and can be scaled separately from storage resources. In some cases, a warehouse includes one or more clusters and / or one or more clusters of server nodes that can work together to provide a service. Applicant has developed and presents herein architectures, methods, algorithms, and systems for multi-cluster warehouses.

[0007] According to one embodiment, a method for a multi-cluster warehouse includes allocating multiple compute clusters as part of a virtual warehouse. The compute clusters may be used to access and execute queries on one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the multiple compute clusters. For example, each of the multiple compute clusters of the virtual warehouse may receive multiple queries such that computing load is distributed to different clusters. The method may also include dynamically adding and removing compute clusters from the virtual warehouse as needed based on the workload of the multiple compute clusters.

[0008] A multi-cluster warehouse can provide significant improvements in concurrency and availability. For example, a warehouse typically contains only a single cluster whose size is the size of the warehouse. For example, a large warehouse may contain a single cluster of eight server nodes. A multi-cluster warehouse may enable the creation of a single warehouse with multiple clusters. Each cluster in a warehouse may contain eight server nodes. Thus, a multi-cluster warehouse may support three times the concurrency level provided by a single cluster warehouse of the same size. This architecture may enable high levels of concurrency for a single warehouse while also allowing for scaling of computing resources, as discussed further herein.

[0009] Improved availability can also be achieved in a multi-cluster warehouse by locating different clusters in different availability zones. For example, because each warehouse cluster is allocated in a different availability zone of a cloud provider (e.g., in a different Amazon® availability zone), a multi-cluster warehouse provides improved fault resilience. Therefore, a multi-cluster warehouse has higher availability compared to a single-cluster warehouse. Furthermore, queries can be routed to the optimal cluster where the relevant data segment already resides in memory or local disk-based storage (e.g., in a cache). For example, a method for a multi-cluster warehouse can include routing queries based on the session in which the query originates. Serving queries from the same session to the same cluster increases the likelihood that the data required for the query already resides in memory, eliminating the need to retrieve the data from cloud storage resources. Due to the increased concurrency and availability, users can experience improved response times and availability that would be difficult or impossible to achieve with other traditional single-cluster database architectures.

[0010] In addition to increased concurrency and availability, significant changes in the automatic scaling of computing resources are possible. For example, at least some embodiments provide for the allocation of computing resources separate from cloud storage. Thus, a multi-cluster warehouse may scale the number of computing clusters to accommodate large fluctuations in workload while still serving queries against data that is static or changing very slowly compared to the query workload.

[0011] Automatic resumption or start of a new or suspended cluster may occur when a warehouse cannot handle the workload and queries must be queued (or queries are queued for longer than the allowed length of time). A query may be queued because the total resource consumption on the cluster exceeds a threshold. For example, resource consumption may include parameters for memory load and computing or processing load. In one embodiment, parameters control how long a query may wait before a new cluster should be resumed or provisioned. As soon as the new cluster is resumed, the query may be scheduled to run on the new cluster. This applies to new queries and queries that are already queued.

[0012] In one embodiment, a method for a multi-cluster warehouse may include dynamically adding compute clusters to a virtual warehouse based on workload. The method may include determining whether a query can be processed while meeting performance metrics for the query. If the query in combination with the current workload does not allow one or more currently allocated compute clusters to meet the performance metrics, the method may include triggering the activation of a new compute cluster. In some embodiments, new clusters may be allocated quickly enough to ensure that no single query falls short of the required performance metrics.

[0013] Automatic suspension or decommissioning of an active cluster of a multi-cluster warehouse may be performed if the resource consumption of the workload is low enough that suspending the cluster would not have caused any previously executed queries to wait N minutes (or wait longer than a threshold time). Query waits or wait times for queries are just one example of a performance metric that may be used. In one embodiment, a method for a multi-cluster warehouse may include removing compute clusters based on workload. The method may include determining whether the current workload can be serviced by fewer compute clusters than the plurality of compute clusters while meeting a performance metric. The method may include decommissioning (or suspending) at least one compute cluster of the plurality of compute clusters in response to determining that the workload can be serviced by fewer compute clusters than the plurality of compute clusters.

[0014] According to one embodiment, the automatic provisioning or removal of clusters and the routing of queries to different clusters within a warehouse can be used as a service and as part of a powerful and flexible multi-cluster warehouse.

[0015] Detailed descriptions of systems and methods consistent with embodiments of the present disclosure are provided below. While several embodiments are described, it should be understood that this disclosure is not limited to any one embodiment, but instead encompasses numerous alternatives, modifications, and equivalents. Also, while numerous specific details are set forth in the following description to provide a thorough understanding of the embodiments disclosed herein, some embodiments may be practiced without some or all of these details. Furthermore, for purposes of clarity, some technical items that are well known in the relevant art have not been described in detail to avoid unnecessarily obscuring the disclosure.

[0016] Turning to the figures, Figure 1 is a block diagram illustrating a processing platform 100 for providing and / or managing a multi-cluster warehouse, according to one embodiment. Processing platform 100 includes a resource manager 102 that is accessible by multiple users 104, 106, and 108. Resource manager 102 may also be referred to herein as a database service manager. In some implementations, resource manager 102 may support any number of users desiring access to data or services of processing platform 100. Users 104-108 may include, for example, end users who provide queries and requests for data storage and retrieval, system administrators who manage the systems and methods described herein, software applications that interact with databases, and other components / devices that interact with resource manager 102.

[0017] The resource manager 102 may provide various services and functions that support the operation of systems and components within the processing platform 100. The resource manager 102 has access to store metadata 110 associated with data stored throughout the data processing platform 100. The resource manager 102 may use the metadata 110 to optimize user queries. In some embodiments, the metadata 110 includes an overview of data stored in remote data storage systems and data available from a local cache (e.g., a cache in one or more clusters of the execution platform 112). The metadata 110 may also include information about how to organize data in the remote data storage systems and local cache. The metadata 110 enables systems and services to determine whether data needs to be processed without loading or accessing the actual data from a storage device.

[0018] As part of the data processing platform 100, metadata 110 may be collected when data is modified using data manipulation language (DML), which may be modified through any DML statement. Examples of data manipulation may include, but are not limited to, selecting, updating, modifying, merging, and inserting data into a table. As part of the processing platform 100, files may be created and metadata 110 may be collected on a per-file and per-column basis, after which the metadata 110 may be stored in a metadata store. This collection of metadata 110 may be performed during data collection, or collection of metadata 110 may be performed as a separate process after data is collected or loaded. In an implementation, the metadata 110 may include distinct values, null values, and minimum and maximum values ​​for each file. In an implementation, the metadata may further include string length information and character ranges within strings.

[0019] The resource manager 102 also communicates with an execution platform 112, which provides multiple computing resources to perform various data storage and data retrieval operations, as discussed in more detail below. The execution platform 112 may include one or more compute clusters that can be dynamically allocated or suspended to a particular warehouse based on the amount of query work provided to that warehouse by users 104-108. The execution platform 112 communicates with one or more data storage devices 116, 118, and 120 that are part of the storage platform 114. Although three data storage devices 116, 118, and 120 are shown in FIG. 1 , the execution platform 112 can communicate with any number of data storage devices. In some embodiments, the data storage devices 116, 118, and 120 are cloud-based storage devices located in one or more geographic locations. For example, the data storage devices 116, 118, and 120 may be part of a public or private cloud infrastructure, or any other type of distributed storage system. Data storage devices 116, 118, and 120 may include hard disk drives (HDDs), solid state drives (SSDs), storage clusters, or any other data storage technology, and storage platform 114 may include a distributed file system (such as the Hadoop Distributed File System (HDFS)), an object storage system, and the like.

[0020] In some embodiments, the communication links between the resource manager 102 and the users 104-108, the metadata 110, and the execution platform 112 are implemented over one or more data communication networks, and may be assigned various tasks so that user requests can be optimized. Similarly, the communication links between the execution platform 112 and the data storage devices 116-120 in the storage platform 114 are implemented over one or more data communication networks. These data communication networks may utilize any communication protocol and any type of communication medium. In some embodiments, a data communication network is a combination of two or more data communication networks (or sub-networks) coupled to each other. In other embodiments, these communication links are implemented using any type of communication medium and communication protocol.

[0021] 1, data storage devices 116, 118, and 120 are separated from the computing resources associated with execution platform 112. This architecture supports dynamic changes to data processing platform 100 based on changing data storage / retrieval, computing, and modification requirements of users and systems accessing data processing platform 100. Supporting dynamic changes allows data processing platform 100 to scale instantly in response to changing demands of systems and components within data processing platform 100. The separation of computing resources from data storage devices supports the storage of large amounts of data without requiring correspondingly large amounts of computing resources. Similarly, this separation of resources supports significant increases in the computing resources utilized at a particular time without requiring a corresponding increase in available data storage resources.

[0022] 1 as individual components. However, each of the resource manager 102, the metadata 110, the execution platform 112, and the storage platform 114 may be implemented as a distributed system (e.g., distributed across multiple systems / platforms in multiple geographic locations) or may be combined into one or more systems. Also, each of the resource manager 102, the storage for the metadata 110, the execution platform 112, and the storage platform 114 may be scaled up or down (independently of each other) based on changing requests received from users 104-108 or changes required by the data processing platform 100. Thus, in the described embodiment, the data processing platform 100 is dynamic and supports regular changes to meet current data processing needs.

[0023] The execution platform 112 includes multiple compute clusters 122, 124, 126 that can share the computational or processing load of the processing platform 100. In one embodiment, a customer can control the number of active (i.e., running) clusters by specifying ranges (e.g., values ​​such as minClusterCount and maxClusterCount) when creating or modifying the configuration of a warehouse (both while the warehouse is running and while it is suspended). A customer can specify the exact number of active clusters by specifying that the warehouse has a precise number to run each time it runs, e.g., by making the minimum cluster count equal to the maximum cluster count. If a user specifies a maximum cluster count greater than the minimum cluster count, the resource manager 102 can automatically manage the number of currently active clusters based on the workload to meet throughput criteria and be cost-effective. Thus, whenever a warehouse is running, at least the minimum cluster count (minClusterCount) of clusters and at most the maximum cluster count (maxClusterCount) of clusters are activated. The resource manager 102 can determine how many clusters are needed to handle the current workload given specified performance criteria in terms of memory load and concurrency level.

[0024] 2 illustrates a block diagram showing components of resource manager 102, according to one embodiment. Resource manager 102 includes an access manager 202 and a key manager 204 coupled to a data storage device 206. Access manager 202 handles authentication and authorization tasks for the system described herein. Key manager 204 manages the storage and authorization of keys used during authentication and authorization tasks. Request processing service 208 manages received data storage and data retrieval requests. Management console service 210 supports access to various systems and processes by administrators and other system managers.

[0025] The resource manager 102 also includes an SQL compiler 212, an SQL optimizer 214, and an SQL executor 216. The SQL compiler 212 parses SQL queries and generates execution code for the queries. The SQL optimizer 214 determines the best way to execute the queries based on the data that needs to be processed. The SQL executor 216 executes the query code for queries received by the resource manager 102, and the query scheduler and coordinator 218 sends received queries to the most appropriate service or system for compilation, optimization, and dispatch to the execution platform 112. The virtual warehouse manager 220 manages the operation of multiple virtual warehouses, including multi-cluster warehouses, implemented on the execution platform 112.

[0026] The resource manager 102 also includes a configuration and metadata manager 222, which manages information about data stored in remote data storage devices and in local caches. A monitor and workload analyzer 224 oversees the processing performed by the resource manager 102 and manages the distribution of tasks (e.g., workload) across virtual warehouses and execution nodes in the execution platform 112. The configuration and metadata manager 222 and the monitor and workload analyzer 224 are coupled to a data storage device 226.

[0027] The resource manager 102 also includes a transaction management and access control module 228, which manages various tasks and other activities associated with the processing of data storage requests and data access requests. For example, the transaction management and access control module 228 provides consistent and synchronized access to data by a number of users or systems. Since a number of users / systems can access the same data simultaneously, changes to the data must be synchronized to ensure that each user / system works with the current version of the data. The transaction management and access control module 228 provides control of various data processing activities at a centralized location within the resource manager 102.

[0028] Referring further to the virtual warehouse manager 220, automatic cluster resume and automatic cluster suspend in a multi-cluster warehouse are discussed. In one embodiment, the virtual warehouse manager 220 performs an automatic cluster resume. When a multi-cluster warehouse (e.g., within the execution platform 112) is marked for automatic resume, the first cluster for that warehouse is automatically resumed when SQL statements are scheduled and all clusters in that warehouse are in a suspended state. However, the decision to automatically resume the remaining clusters is made based on the workload. This is believed to be activeClusterCount < maxClusterCount, that is, having clusters that can be active / resumed but are currently suspended.

[0029] Workload considerations include at least two things. First, workload considerations can include memory usage. If a query is scheduled and waits because all clusters are at their maximum memory capacity, virtual warehouse manager 220 resumes one or more clusters so that the wait can be avoided or shortened. Because resuming a cluster can take a short time, e.g., minutes, a wait can still occur if a new cluster needs to be resumed. However, virtual warehouse manager 220 can also ensure that there is a free pool of some free servers so that queries can be placed on the free pool while starting a new cluster. Virtual warehouse manager 220 can also wait a period of time to see if the query resolves itself before deciding to provision a new cluster.

[0030] Second, workload considerations may include the concurrency, or processing / computing load, for a cluster. If concurrency on all active clusters is high, the virtual warehouse manager 220 may start another cluster even if there is enough memory to schedule queries. Here, concurrency may be calculated for each active cluster based on the degree of parallelism (DOP). Specifically, concurrency may be the number of queries running at the total DOP. For example, this may be calculated by dividing the DOP for all running queries by the product of the maximum DOP (MAX_DOP) and the number of running queries. This may be a fractional or non-integer number, as some lightweight queries run at a DOP smaller than the maximum. In one embodiment, a warehouse parameter may be specified to control concurrency. For example, the concurrency (concurrency_level_target) may be set to a value of 8 by default. This parameter may be exposed to the customer, as its value actually depends on how much the customer is willing to spend on the problem and how much query performance they are willing to degrade (compared to standalone query performance) when the warehouse is shared.

[0031] In one embodiment, the virtual warehouse manager 220 performs automatic cluster suspension. In one embodiment, after a certain number of seconds of inactivity (e.g., based on the auto_suspend parameter), the entire warehouse may shut down. Alternatively, if a warehouse has two or more active clusters, one or more clusters may be suspended if the warehouse has been running below its capacity for longer than a predetermined time, measured in minutes, for example. For example, consider a warehouse with three active clusters. If the warehouse is under load, i.e., has been able to execute all SQL statements issued by the engine for the current time without waiting or proceeding beyond maximum concurrency, for longer than a predetermined time, one or more clusters may be suspended. Note that a warehouse utilizes all currently active clusters while under load. Clusters do not need to be inactive for a certain period of time to shut down. Checks for automatic cluster suspension may be performed periodically, such as 5 minutes remaining, 10 minutes remaining, 30 minutes remaining, 1 hour remaining, etc. In one embodiment, checks for automatic cluster suspension may be performed at intervals other than the predetermined time. For example, a check to see if there were 10 minutes under load remaining can be performed at each time change so that customers can be billed by the hour.

[0032] With further reference to the query scheduler and coordinator 218, query scheduling may be performed based on workload, query similarity, and other factors. The query scheduler and coordinator 218 may forward queries to specific clusters based on workload. For example, the query scheduler and coordinator 218 may attempt to maintain approximately equal workload on each cluster to spread processing tasks and improve query execution time and user experience. Query similarity may be used to ensure that related queries, particularly queries related to the same data, are sent to the same cluster. For example, the query scheduler and coordinator 218 may send queries with the same session identifier to the same cluster. Forwarding queries based on query similarity may enable the query scheduler and coordinator 218 to ensure that data for a scheduled query is already present in the local cache of a specific cluster. This may significantly reduce response time, workload, and data retrieval.

[0033] FIG. 3 is a schematic block diagram illustrating a multi-cluster warehouse 302 and the scheduling of queries 304 on the multi-cluster warehouse 302. The warehouse 302 includes multiple clusters (Cluster 1, Cluster 2, Cluster N), each including multiple server nodes. In one embodiment, each cluster includes the same number of servers, although this may vary in different embodiments. In one embodiment, each server in a cluster belongs to the same availability zone, although different clusters may be located in different availability zones. The concept of warehouse availability may be based on the overall availability percentage of the warehouse. For example, availability for a particular cluster within the warehouse 302 may be the percentage of servers that are available (e.g., operational) compared to the cluster size. However, if that percentage is below the minimum required to execute a query (e.g., 50%), availability may be determined to be 0% for that cluster, and no queries may be assigned to the cluster until the warehouse 302 or some servers in the warehouse 302 are repaired. As discussed herein, the number of clusters in warehouse 302 may be dynamically adjusted based on workload, cluster server failures, and the like.

[0034] In one embodiment, the query scheduler and coordinator 218 weights each query (e.g., an SQL statement or portion of an SQL statement) based on its planned resource consumption. For example, some queries may take significantly more memory to execute, while other queries may take significantly more processing resources to execute. Similarly, some queries have high or low consumption for both memory and processing. The resource manager 102 may determine what the predicted or planned consumption is and may then be able to determine where to place the query to most effectively balance the workload among different clusters. For example, a high-consumption query may use as many resources as many low-consumption queries.

[0035] In one embodiment, the query scheduler and coordinator 218 may schedule a query on one or more clusters in the warehouse 302, or may wait for a query if the workload is too high or availability is too low. For example, the query scheduler and coordinator 218 may first attempt to schedule the query 304 (e.g., an SQL statement) on an active (i.e., not suspended) cluster in the warehouse 302. If there are multiple active clusters, the query scheduler and coordinator 218 may eliminate a set of clusters on which the query 304 will not execute because they are not available or memory may be oversubscribed. As previously mentioned, a cluster may be determined to be unavailable by default if less than 50% of the servers in a node are unavailable (e.g., have failed). If multiple possible clusters remain, the query scheduler and coordinator 218 may choose the cluster with the lowest load, which in one embodiment is defined as the sum of the DOPs of all jobs running on that cluster. The cluster with the lowest load may also be based on the sum of all memory needs for that cluster. If there are multiple clusters with equal load, the query scheduler and coordinator 218 may use the session ID for a particular query 304 as a tiebreaker so that queries from the same session can run on the same cluster. The queries 304 that are assigned to clusters in the warehouse 302 are shown as running queries 306.

[0036] If no cluster exists for scheduling a particular query, the query scheduler and coordinator 218 may queue the query in a global queue. A globally queued query 304 is shown as a queued query 308. A queued query 308 may remain queued until one of the clusters in the warehouse 302 becomes free or available. Note that one or more servers in an assigned cluster may be marked as suspected of failure if some running queries 306 must wait for the cluster to be repaired.

[0037] FIG. 4 is a block diagram illustrating an embodiment of multiple warehouses that may be active or operating on a single execution platform 112, according to one embodiment. Multiple virtual warehouses 402, 404, 406 are shown, with each virtual warehouse including multiple clusters 408. Each cluster 408 includes multiple execution nodes 410, each including a processor 412 and a cache 414 (e.g., memory). While three virtual warehouses 402-406 are shown, the number of virtual warehouses may change dynamically. Similarly, the number of clusters 408 in each warehouse 402-406 and the number of execution nodes 410 in each cluster may vary in different embodiments, and may vary indefinitely relative to one another. Furthermore, the number of clusters 408 in a virtual warehouse and the number of execution nodes 410 in a cluster may be dynamic, such that new clusters 408 and execution nodes 410 may be created or removed as demand changes.

[0038] Each virtual warehouse 402-406 can access any of the data storage devices 116-120 shown in Figure 1. Thus, the virtual warehouses 402-406 need not be assigned to a particular data storage device 116-120, but instead may access data from any of the data storage devices 116-120. Similarly, each of the clusters 408 and execution nodes 410 may access data from any of the data storage devices 116-120. In some embodiments, a particular virtual warehouse or execution node may be temporarily assigned to a particular data storage device, but the virtual warehouse or execution node may later access data from any other data storage device.

[0039] While the illustrated execution nodes 410 each include one cache and one processor, other embodiments may include execution nodes including any number of processors and any number of caches. Also, caches may vary in size between different execution nodes 410. The cache 414 stores data retrieved from one or more data storage devices in the storage platform 114 (FIG. 1) at the local execution node. Thus, the cache reduces or eliminates bottleneck problems that arise in platforms that consistently retrieve data from remote storage systems. Instead of repeatedly accessing data from remote storage devices, the systems and methods described herein access data from a cache in the execution node, which is significantly faster and avoids the bottleneck problems discussed above. In some embodiments, the cache is implemented using high-speed memory devices, which provide fast access to cached data. Each cache can store data from any storage device in the storage platform 114.

[0040] Furthermore, cache and computing resources may vary between different execution nodes. For example, one execution node may include significant computing resources and minimal cache resources, making the execution node useful for tasks requiring significant computing resources. Another execution node may include significant cache resources and minimal computing resources, making the execution node useful for tasks requiring caching of large amounts of data. In some embodiments, the cache and computing resources associated with a particular execution node are determined at the time the execution node is created based on the anticipated tasks to be performed by the execution node.

[0041] Additionally, the cache and computing resources associated with a particular execution node may change over time based on changes in the tasks performed by the execution node. For example, if the tasks performed by an execution node are more processor-intensive, the particular execution node may be assigned more processors. Similarly, if the tasks performed by an execution node require more cache capacity, the execution node may be assigned more cache resources.

[0042] Although virtual warehouses 402-406 are associated with the same execution platform 112, the virtual warehouses may be implemented using multiple computing systems in multiple geographic locations. For example, virtual warehouse 402 may be implemented by a computing system in a first geographic location, while virtual warehouses 404 and 406 are implemented by different computing systems in a second geographic location. In some embodiments, these different computing systems are cloud-based computing systems maintained by one or more different entities.

[0043] Each virtual warehouse is also depicted in FIG. 4 as having multiple clusters 408. The clusters 408 associated with each virtual warehouse may be implemented using multiple computing systems in multiple geographic locations or in different availability zones. For example, a particular instance of a virtual warehouse 402 may implement a cluster 408 with execution nodes 410 on one computing platform in a particular geographic location, and other clusters 408 and execution nodes 410 on different computing platforms in another geographic location. The virtual warehouses 402-406 are also fault-tolerant. For example, a given virtual warehouse or execution node 410 can be quickly repaired in the same or a different geographic location.

[0044] A particular execution platform 112 may include any number of virtual warehouses 402-406. Additionally, the number of virtual warehouses in a particular execution platform is dynamic, such that new virtual warehouses are created when additional processing and / or cache resources are needed. Similarly, existing virtual warehouses may be deleted when the resources associated with those virtual warehouses are no longer needed.

[0045] 5 illustrates a block diagram of another exemplary operating environment 500 having multiple distributed virtual warehouses and execution platforms. Environment 500 includes a resource manager 102 that communicates with execution platforms 1 504 and execution platforms 2 506 over a data communications network 502. Execution platforms 1 504 includes two clusters, specifically, cluster A for a first virtual warehouse 508 and cluster A for a second virtual warehouse 510. Execution platforms 2 506 includes two additional clusters, specifically, cluster B for a first virtual warehouse 514 and cluster B for a second virtual warehouse 516. Resource manager 102 also communicates over data communications network 502 with cluster C of a first virtual warehouse 512 (which is not part of either execution platforms 504, 506).

[0046] Execution platforms 504 and 506 and cluster C for first virtual warehouse 512 communicate with databases 520, 522, and 524 through data communications network 518. In some embodiments, data communications networks 502 and 518 are the same network or a combination of one or more overlapping networks. Environment 500 enables resource manager 102 to coordinate user data storage and retrieval requests across multiple clusters 508-516 of multiple warehouses to store or retrieve data in databases 520-524. Execution platforms 504 and 506 and cluster C for first virtual warehouse 512 may be located in the same or different geographic regions or in the same or different availability zones. Additionally, execution platforms 504 and 506 may be implemented by the same entity or by different entities.

[0047] The systems and methods described herein allow data to be stored and accessed as a service separate from computing (or processing) resources. Data is available to the virtual warehouse without requiring reloading from a remote data source, even if no computing resources are required from the execution platform. The described systems and methods are useful for any type of data. In certain embodiments, data is stored in a structured, optimal format. The separation of data storage / access services from computing services also simplifies the sharing of data among different users and groups. As discussed herein, each virtual warehouse can access any data to which it has access permissions, even while other virtual warehouses are accessing the same data. This architecture supports executing queries without any actual data stored in a local cache. The systems and methods described herein enable transparent dynamic data movement, which moves data as needed from remote storage devices to a local cache in a manner transparent to users of the system. Furthermore, because any virtual warehouse can access any data due to the separation of data storage services from computing services, this architecture supports the sharing of data without traditional data movement.

[0048] Additionally, environment 500 enables spreading of a single virtual warehouse across multiple geographic locations or availability zones. For example, clusters 508, 512, and 514 may all belong to the same virtual warehouse (first virtual warehouse) but be located in different geographic regions or availability zones. Because outages or failures can occur across geographic regions or availability zones, improved fault tolerance may be achieved. For example, availability zones may be implemented by cloud service (compute or storage) providers so that there is little or no chance that an issue in one availability zone will spread to a different availability zone. Thus, clusters within the same warehouse but in different availability zones may significantly reduce the chance that a warehouse will be left without any available execution or compute nodes.

[0049] In one embodiment, the multi-cluster warehouse embodiments disclosed herein may use a specialized Data Definition Language (DDL). The following are some examples of commands or instructions that may be part of the DDL for a multi-cluster warehouse: □ create warehouse single_cluster size=xlarge; / / this will create a single cluster warehouse □ create warehouse multi_cluster size=xlarge max_cluster_count=3 min_cluster_count=1; / / this will create an x-large 3 cluster warehouse. Only one cluster will be started by default □ create warehouse multi_cluster size=xlarge max_cluster_count=3 min_cluster_count=2; / / this will create an x-large warehouse with 2 clusters initially resumed □ create warehouse multi_cluster size=xlarge max_cluster_count=3 min_cluster_count=3; / / this will create an x-large warehouse with all clusters resumed □ Note that the resource manager would try to make use of all availability zones, one per cluster. The availability zone to use for each cluster may be implemented by an infrastructure management system □ alter warehouse <warehouse_name> set warehouse_size= <size>: allows one to change the size of the warehouse. If this warehouse is started, all clusters in the warehouse will be resized. The code to implement this instruction may include a resize operation for each cluster. □ alter warehouse <warehouse_name> set max_cluster_count= <count>this will add or remove clusters from an existing warehouse. Internally clusters may be numbered so this operation will either add new clusters at the end of the range or remove clusters starting from the end of the range. If new clusters are created, they will be created in a suspended state. If clusters are removed and these clusters are active, they will first be inactivated (quiesced) to allow running queries to terminate. □ Drop warehouse<warehouse_name> drop warehouse and all associated clusters. Clusters will be inactivated (quiesced) before dropping them.

[0050] Returning to the drawing, Figure 6 is a schematic flow chart diagram illustrating an exemplary method 600 for a multi-cluster warehouse. Method 600 may be performed by a processing platform or resource manager, such as processing platform 100 of Figure 1 or a resource manager of Figures 1, 2, or 5.

[0051] Method 600 begins with a system allocating (602) multiple compute clusters as part of a virtual warehouse to access and execute queries against one or more databases in one or more cloud storage resources. In one embodiment, the multiple compute clusters are allocated by the system independently of one or more cloud storage resources, such that the number of compute clusters can expand or contract without increasing or decreasing the one or more cloud storage resources. The system provides (604) queries against the virtual warehouse to each of the multiple compute clusters. For example, multiple queries may be provided to each of the multiple compute clusters in the virtual warehouse. The system dynamically adds and removes compute clusters from the virtual warehouse as needed based on the workload of the multiple compute clusters. Method 600 may also include determining the workload for the multiple compute clusters. The system may determine the workload by determining the availability of one or more processor resources for each of the multiple compute clusters and memory resources for each of the multiple compute clusters.

[0052] Method 600 may be implemented by a database system or device such that a single entity, such as a warehouse, can grow or shrink depending on the number of queries. Specifically, a resource manager or other system may allow a warehouse to grow or shrink as changes in the concurrency (or computational and memory load) of the warehouse occur.

[0053] Figure 7 is a schematic flow chart diagram illustrating an exemplary method 700 for dynamically adding compute clusters in a multi-cluster warehouse. Method 700 may be performed by a processing platform or resource manager, such as processing platform 100 of Figure 1 or a resource manager of Figures 1, 2, or 5. Method 700 may be performed in combination with or separately from method 600 of Figure 6.

[0054] Method 700 begins with the system determining 702 whether a query can be processed while meeting performance metrics for the query. In one embodiment, method 700 includes determining 702 whether a query can be processed for each query directed to a compute cluster such that performance metrics for each query are met. The performance metrics may include a service level agreement (SLA) acceptable to a customer. For example, the SLA may require that a query be scheduled within a specific time (e.g., 10 seconds). This may limit any query from waiting in a global queue for more than a maximum time (e.g., 10 seconds). The SLA may be agreed upon in advance between the warehouse as a service provider and the customer. Different price tiers may be offered based on the SLA, or the SLA may specify that the system will use more resources to ensure users experience minimal delays in accessing and executing queries against the database.

[0055] In response to determining that one or more currently allocated compute clusters cannot accommodate the query combined with the current workload to meet the performance metrics, the system triggers activation of a new compute cluster (704). In one embodiment, the system may trigger activation (704) only if the number of currently active clusters is less than a predetermined maximum number of compute clusters.

[0056] Figure 8 is a schematic flow chart diagram illustrating an exemplary method 800 for dynamically removing a compute cluster in a multi-cluster warehouse. Method 800 may be performed by a processing platform or resource manager, such as processing platform 100 of Figure 1 or a resource manager of Figures 1, 2, or 5. Method 800 may be performed in combination with one or more of methods 600 or 700 of Figures 6 and 7, or separately from one or more of methods 600 or 700.

[0057] Method 800 begins with the system determining 802 whether a current workload can be serviced by fewer than a plurality of compute clusters while meeting performance metrics. In one embodiment, determining 802 whether a current workload can be serviced by fewer than a plurality of compute clusters may include determining whether historical workload over a period leading up to the present time could have been serviced by fewer than a plurality of compute clusters while meeting performance metrics. For example, if the best cluster were removed from the virtual warehouse, would the virtual warehouse have been able to process all queries while meeting performance metrics?

[0058] The system decommissions (804) (or renders inactive) at least one compute cluster of the plurality of compute clusters in response to determining that the workload is serviceable by fewer compute clusters than the plurality of compute clusters. The system may decommission (804) or remove a compute cluster only if the current number of active clusters is less than a predetermined minimum number of compute clusters. In one embodiment, decommissioning (804) at least one compute cluster may include decommissioning in response to determining that the historical workload for the period was serviceable by fewer compute clusters than the plurality of compute clusters.

[0059] In one embodiment, decommissioning 804 the at least one compute cluster includes quiescing the at least one compute cluster to prevent it from serving or scheduling additional queries. Decommissioning 804 may also include allowing the at least one compute cluster to complete currently assigned queries and releasing one or more resources corresponding to the at least one compute cluster upon completion of already scheduled or active queries.

[0060] Figure 9 is a schematic flow chart diagram illustrating an exemplary method 900 for a multi-cluster warehouse. Method 900 may be performed by a processing platform or resource manager, such as processing platform 100 of Figure 1 or a resource manager of Figures 1, 2, or 5. Method 900 may be performed in combination with one or more of methods 600, 700, or 800 of Figures 6, 7, and 8, or separately from one or more of methods 600, 700, or 800.

[0061] Method 900 begins with a system allocating (902) multiple compute clusters as part of a virtual warehouse to access and execute queries against one or more databases in one or more cloud storage resources. The system then forwards (904) queries against the virtual warehouse to each of the multiple compute clusters. The multiple queries may be provided to each of the multiple compute clusters of the virtual warehouse. In one embodiment, forwarding (904) queries against the virtual warehouse to each of the multiple compute clusters includes routing (906) queries based on the session in which the query originated, such that queries from the same session are routed to the same compute cluster by default. Each cluster has the ability to persist fragments of the databases it processes. That is, each cluster (or each compute node in a cluster) may maintain a cache of all tables currently accessed while processing queries on the cluster. Thus, a resource manager or scheduler may move queries from the same query stream (e.g., with the same session identifier) ​​to the same cluster to take advantage of caching. In some cases, if the cluster handling a particular session has significantly fewer resources available than another cluster, queries with the same session identifier may end up on different clusters.

[0062] In one embodiment, the system may route the query based on the workload of each of the multiple compute clusters (906). For example, if a cluster cannot accept new queries, the system may offer the query to a different cluster even if the different cluster has not processed a query corresponding to the same session. In one embodiment, the system may offer the query to at least two compute clusters in different availability zones (904).

[0063] FIG. 10 is a block diagram illustrating an exemplary computing device 1000. In some embodiments, the computing device 1000 may be used to implement one or more systems and components discussed herein. For example, the computing device 1000 may allow a user or administrator to access the resource manager 102. Alternatively, a component, system, or platform discussed herein may include one or more computing devices 1000. Furthermore, the computing device 1000 may interact with any of the systems and components described herein. Thus, the computing device 1000 may be used to perform various procedures and tasks, such as those discussed herein. The computing device 1000 may function as a server, a client, or any other computing entity. The computing device 1000 may be any of a wide variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a portable computer, and a tablet.

[0064] Computing device 1000 includes one or more processors 1002, one or more memory devices 1004, one or more interfaces 1006, one or more mass storage devices 1008, and one or more input / output devices 1010, all coupled to a bus 1012. Processor 1002 includes one or more processors or controllers that execute instructions stored in memory device(s) 1004 and / or mass storage device(s) 1008. Processor 1002 may also include various types of computer-readable media, such as cache memory.

[0065] The memory device 1004 includes a variety of computer-readable media, such as volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read-only memory (ROM)). The memory device 1004 may also include re-writable ROM, such as flash memory.

[0066] The mass storage device 1008 includes a variety of computer-readable media, such as magnetic tape, magnetic disks, optical disks, and solid-state memory (e.g., flash memory). The mass storage device 1008 may also include a variety of drives to allow reading from and / or writing to the various computer-readable media. The mass storage device 1008 includes removable media and / or non-removable media.

[0067] The input / output devices 1010 include a variety of devices that allow data and / or other information to be input to or retrieved from the computing device 1000. Exemplary input / output devices 1010 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, and CCD or other image capture devices, etc.

[0068] Interface 1006 includes various interfaces that allow computing device 1000 to interact with other systems, devices, or computing environments. Exemplary interface 1006 includes any number of different network interfaces, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet.

[0069] The bus 1012 allows the processor 1002, memory device 1004, interface 1006, mass storage device 1008, and input / output device 1010 to communicate with each other and with other devices or components coupled to the bus 1012. The bus 1012 represents one or more of various types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, and a USB bus. (example)

[0070] The following examples relate to further embodiments.

[0071] Example 1 is a computer-implemented method for a multi-cluster warehouse. The method includes allocating multiple compute clusters as part of a virtual warehouse to access and execute queries against one or more databases in one or more cloud storage resources. The method includes providing queries against the virtual warehouse to each of the multiple compute clusters, where the multiple queries are provided to each of the multiple compute clusters of the virtual warehouse. The method includes dynamically adding and removing compute clusters from the virtual warehouse as needed based on workloads of the multiple compute clusters.

[0072] In Example 2, the multiple computing clusters of Example 1 are allocated independently of one or more cloud storage resources such that the number of computing clusters can be expanded or contracted without increasing or decreasing one or more cloud storage resources.

[0073] In Example 3, the method of any of Examples 1-2 further includes determining workload for a plurality of compute clusters. Determining workload includes determining availability of one or more processor resources for each of the plurality of compute clusters and memory resources for each of the plurality of compute clusters.

[0074] In Example 4, dynamically adding a compute cluster in any of Examples 1-3 includes determining whether the query can be processed while meeting performance metrics for the query, and triggering activation of a new compute cluster in response to determining that the query in combination with the current workload does not allow one or more currently assigned clusters to meet the performance metrics.

[0075] In Example 5, the method of Example 4 includes determining, for each query directed to a compute cluster such that a performance metric is met for each query, whether the query can be processed.

[0076] In Example 6, the performance metric in any of Examples 4-5 includes a service level agreement accepted by a customer.

[0077] In Example 7, the performance metric in any of Examples 4-6 includes a maximum period of time that a query will be waited.

[0078] In Example 8, dynamically adding computing clusters in any of Examples 1 to 7 includes adding computing clusters up to a predetermined maximum number of computing clusters.

[0079] In Example 9, dynamically removing compute clusters in any of Examples 1-8 includes removing compute clusters until a predetermined minimum number of compute clusters is reached.

[0080] In Example 10, removing a compute cluster in any of Examples 1-9 includes determining whether a current workload can be serviced by fewer compute clusters than the plurality of compute clusters while meeting a performance metric, and decommissioning at least one compute cluster of the plurality of compute clusters in response to determining that the workload can be serviced by fewer compute clusters than the plurality of compute clusters.

[0081] In Example 11, decommissioning the at least one compute cluster in Example 10 includes preventing additional queries from being submitted to the at least one compute cluster, allowing the at least one compute cluster to complete currently assigned queries, and releasing one or more resources corresponding to the at least one compute cluster upon completion of the currently assigned queries.

[0082] In Example 12, determining whether the current workload of any of Examples 10-11 can be serviced by fewer than the plurality of compute clusters includes determining whether the historical workload for the time period leading up to the current time could have been serviced by fewer than the plurality of compute clusters while meeting a performance metric. Retiring at least one compute cluster includes retiring in response to determining that the historical workload for the time period could have been serviced by fewer than the plurality of compute clusters.

[0083] In Example 13, providing the query for the virtual warehouse to each of the plurality of compute clusters in any of Examples 1-12 includes routing the query based on the session in which the query originated.

[0084] In Example 14, providing the query for the virtual warehouse to each of the plurality of compute clusters in any of Examples 1-13 includes routing the query based on the workload of each of the plurality of compute clusters.

[0085] In Example 15, allocating the plurality of computing clusters in any of Examples 1-14 includes allocating at least two computing clusters in different availability zones.

[0086] Example 16 is an apparatus including means for carrying out a method such as any of Examples 1-15.

[0087] Example 17 is a machine-readable storage device containing machine-readable instructions that, when executed, implement the method of any of Examples 1-16 or realize the apparatus of any of Examples 1-16.

[0088] The flow diagrams and block diagrams herein illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function(s). It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium that can instruct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture that includes instruction means for implementing the functions / acts specified in one or more blocks of the flow diagrams and / or block diagrams.

[0089] The systems and methods described herein provide flexible and scalable data warehouses using novel data processing platforms, methods, systems, and algorithms. In some embodiments, the described systems and methods leverage a cloud infrastructure that supports cloud-based storage and computing resources, etc. Exemplary cloud-based storage resources provide significant storage capacity that is available on demand at low cost. Furthermore, these cloud-based storage resources can be fault-tolerant and highly scalable, which can be costly to achieve with private data storage systems. Exemplary cloud-based computing resources are available on demand and can be priced based on actual usage levels of resources. Typically, cloud infrastructures are rapidly and dynamically evolved, provisioned, and decommissioned.

[0090] In the described systems and methods, the data storage system utilizes an SQL (Structured Query Language) based relational database. However, these systems and methods are applicable to any kind of database, using any data storage architecture and using any language for storing and retrieving data within the database. The systems and methods described herein may also provide a multi-tenant system that supports the separation of computing resources and data between different customers / clients and between different users within the same customer / client.

[0091] The various techniques, or some aspects or portions thereof, may take the form of program code (e.g., instructions) embodied in a tangible medium, such as a floppy disk, a CD-ROM, a hard drive, a non-transitory computer-readable storage medium, or any other machine-readable storage medium, where, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for performing the various techniques. In the case of a program executing code on a programmable computer, the computing device may include a processor, a processor-readable storage medium (including volatile and nonvolatile memory and / or storage elements), at least one input device, and at least one output device. The volatile and nonvolatile memory and / or storage elements may be RAM, EPROM, a flash drive, an optical drive, a magnetic hard drive, or another medium for storing electronic data. One or more programs that may implement or utilize the various techniques described herein may use application programming interfaces (APIs), reusable controls, and the like. Such programs may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, the programs may also be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.

[0092] It should be understood that many of the functional units described in this specification can be implemented as one or more components, a term used to more specifically emphasize their implementation independence. For example, components can be implemented as hardware circuits including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Components can also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, or programmable logic devices.

[0093] Components may also be implemented in software for execution by various types of processors. An identified component of executable code may, for example, comprise one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, the executable of an identified component need not be physically located with respect to each other, but may comprise heterogeneous instructions stored in different locations that, when logically coupled to each other, comprise the component and achieve the specified purpose of the component.

[0094] Indeed, a component of executable code may be a single instruction or many instructions, and may even be distributed across several different code segments in different programs. Similarly, operational data may be identified and described herein within components and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or distributed across different locations, including across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network. Components may be passive or active and include agents operable to perform a desired function.

[0095] References throughout this specification to an "example" mean that the particular feature, structure, or characteristic described in connection with the example is included in at least one embodiment of the disclosure. Thus, the appearances of the phrase "in an example" in various places throughout this specification do not necessarily all refer to the same embodiment.

[0096] As used herein, multiple items, structural elements, compositional elements, and / or materials may be presented in common lists for convenience. However, these lists should be construed as though each element of the list is independently identified as a separate and unique element. Accordingly, individual elements of such lists should in no way be construed as de facto equivalents of any other elements of the same list merely based on their presentation within a common grouping without indication to the contrary. Also, various embodiments and examples of the present disclosure may be referred to herein along with alternatives for their various components. It is understood that such embodiments, examples, and alternatives should not be construed as de facto equivalents of each other, but as separate and autonomous representations of the present disclosure.

[0097] Although some details have been set forth above for clarity, some changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatus described herein. Thus, the present embodiments should be considered illustrative and not limiting.

[0098] Those skilled in the art will recognize that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the disclosure, the scope of which should therefore be determined solely by the claims which follow.< / count> < / size>

Claims

1. means for allocating a plurality of compute clusters as part of a virtual warehouse for accessing and querying one or more databases in one or more cloud storage resources located on a storage platform separate from an execution platform, the plurality of compute clusters being allocated separately from the one or more cloud storage resources, each of the plurality of compute clusters including one or more execution nodes including a cache memory for caching data stored on the storage platform; means for determining the current workload of the plurality of compute clusters based on: (i) calculating, for each active compute cluster, a runtime concurrency defined as a normalized value equal to the sum of the degrees of parallelism (DOP) of all queries executing on that compute cluster divided by the product of the maximum DOP (MAX_DOP) and the number of queries executing; and (ii) comparing the runtime concurrency for each active compute cluster to a target concurrency specified as a warehouse parameter presented to a customer; means for dynamically adding a first compute cluster to the virtual warehouse and deactivating a second compute cluster from the virtual warehouse based on whether one or more currently deployed compute clusters can process one or more of the queries and the current workload while satisfying a performance metric, wherein the performance metric comprises a maximum wait time for the query; the means for adding the first compute cluster or deactivating the second compute cluster does not increase or decrease the one or more cloud storage resources; the addition of the first compute cluster is triggered when the runtime concurrency of all active compute clusters exceeds the target concurrency; queries are temporarily scheduled to a free pool of servers during startup of the first compute cluster to respect the maximum wait time; and the deactivation of the second compute cluster comprises: preventing additional queries from being submitted to the second compute cluster; and allowing the second compute cluster to complete currently scheduled queries; releasing one or more resources corresponding to the second compute cluster upon completion of the currently scheduled query. The means comprising: Including, the system.

2. The means for dynamically adding the first compute cluster to the virtual warehouse and deactivating the second compute cluster from the virtual warehouse comprises: means for triggering activation of the first compute cluster in response to determining that one or more currently allocated compute clusters cannot process the query in combination with the current workload while meeting the performance metrics; means for determining whether the current workload can be serviced by one less than the plurality of compute clusters while satisfying the performance metric; and deactivating the second compute cluster is further responsive to determining that the current workload can be serviced by one compute cluster less than the plurality of compute clusters while satisfying the performance metric. The system of claim 1 .

3. and means for providing a query for the virtual warehouse to each of the plurality of compute clusters, the means for providing the query comprising: means for routing queries based on the session from which the queries originate; or means for routing the query based on availability of cluster resources to execute the query; The system of claim 1 , comprising one or more of:

4. allocating a plurality of computational clusters on the execution platform as part of a virtual warehouse for accessing and performing queries on one or more databases in one or more cloud storage resources located on a storage platform separate from the execution platform, the plurality of computational clusters allocated separately from the one or more cloud storage resources, each of the plurality of computational clusters including one or more execution nodes including a cache memory for caching data stored on the storage platform; (i) for each active compute cluster, calculating a runtime concurrency, defined as a normalized value equal to the sum of the degrees of parallelism (DOP) of all queries executing on that compute cluster divided by the product of the maximum DOP (MAX_DOP) and the number of queries executing on that compute cluster; and (ii) determining the current workload of the plurality of compute clusters based on comparing the runtime concurrency for each active compute cluster to a target concurrency specified as a warehouse parameter presented to a customer; dynamically adding a first compute cluster to the virtual warehouse or deactivating a second compute cluster from the virtual warehouse based at least in part on whether one or more currently allocated compute clusters can process one or more of the queries and the current workload while satisfying a performance metric, wherein the performance metric comprises a maximum wait time for the query, adding the first compute cluster or deactivating the second compute cluster without increasing or decreasing the one or more cloud storage resources, adding the first compute cluster is triggered when the runtime concurrency of all active compute clusters exceeds the target concurrency, and during startup of the first compute cluster, queries are temporarily scheduled to a free pool of servers to respect the maximum wait time, and deactivating the second compute cluster preventing additional queries from being submitted to the second compute cluster; and allowing the second compute cluster to complete currently scheduled queries; releasing one or more resources corresponding to the second compute cluster upon completion of the currently scheduled query. Including A method comprising:

5. determining the workload for the plurality of compute clusters, wherein determining the workload includes determining availability of one or more of processor resources for each of the plurality of compute clusters or memory resources for each of the plurality of compute clusters; determining, for each query directed to the compute cluster, whether the query can be processed such that the performance metric is met for each query; The method of claim 4 , further comprising one or more of:

6. Dynamically adding the first compute cluster to the virtual warehouse based on the workload includes: determining whether the query can be processed while satisfying a performance metric for the query; triggering activation of the first compute cluster in response to determining that the query in combination with a current workload will not allow one or more currently assigned compute clusters to meet the performance metric. The method of claim 4, comprising:

7. The method of claim 6 , wherein the performance metrics further include a service level agreement that defines a maximum wait time for the query.

8. 5. The method of claim 4, wherein deactivating the second compute cluster based on the workload comprises determining whether a current workload can be serviced by one compute cluster less than the plurality of compute clusters while satisfying a performance metric.

9. 9. The method of claim 8, wherein determining whether the current workload can be serviced by one compute cluster less than the plurality of compute clusters further comprises determining whether historical workload for a period leading up to the current time could have been serviced by one compute cluster less than the plurality of compute clusters while satisfying the performance metric.

10. Deactivating the second compute cluster includes: deactivating the second compute cluster in response to determining that the historical workload for the time period was serviceable by one compute cluster less than the plurality of compute clusters; or Allocating the plurality of compute clusters includes allocating at least two compute clusters to different availability zones. The method of claim 9 further comprising:

11. Providing a query to the virtual warehouse to each of the plurality of compute clusters comprises: Routing queries based on the session from which the query originates; or The method of claim 4 , comprising one or more of: routing queries based on the workload of each of the plurality of compute clusters.

12. When executed by a data processing device, allocating a plurality of compute clusters on the execution platform as part of a virtual warehouse for accessing and querying one or more databases in one or more cloud storage resources located on a storage platform separate from the execution platform, the plurality of compute clusters allocated separately from the one or more cloud storage resources, each of the plurality of compute clusters including one or more execution nodes including a cache memory for caching data stored on the storage platform; (i) for each active compute cluster, calculating a runtime concurrency, defined as a normalized value equal to the sum of the degrees of parallelism (DOP) of all queries executing on that compute cluster divided by the product of the maximum DOP (MAX_DOP) and the number of queries executing on that compute cluster; and (ii) determining the current workload of the plurality of compute clusters based on comparing the runtime concurrency for each active compute cluster to a target concurrency specified as a warehouse parameter presented to a customer; dynamically adding a first compute cluster to the virtual warehouse and deactivating a second compute cluster from the virtual warehouse based at least in part on whether one or more currently allocated compute clusters can process one or more of the queries and the current workload while satisfying a performance metric, wherein the performance metric comprises a maximum wait time for the query, adding the first compute cluster or deactivating the second compute cluster without increasing or decreasing the one or more cloud storage resources, adding the first compute cluster is triggered when the runtime concurrency of all active compute clusters exceeds the target concurrency, and during startup of the first compute cluster, queries are temporarily scheduled to a free pool of servers to respect the maximum wait time, and deactivating the second compute cluster preventing additional queries from being submitted to the second compute cluster; and allowing the second compute cluster to complete currently scheduled queries; releasing one or more resources corresponding to the second compute cluster upon completion of the currently scheduled query. Including A non-transitory readable storage medium storing instructions that cause the data processing apparatus to perform operations including:

13. The operation is allocating the plurality of computing clusters independently of the one or more cloud storage resources such that the number of computing clusters can be scaled up and down without increasing or decreasing the one or more cloud storage resources; determining the workload for the plurality of compute clusters by determining the availability of one or more of processor resources for each of the plurality of compute clusters, memory resources for each of the plurality of compute clusters, or predicting the minimum resources required to process a particular query; or determining whether a query can be processed while meeting a performance metric for the query, and dynamically adding a first compute cluster to the virtual warehouse based on the workload by triggering activation of a first compute cluster in response to determining that the query, in combination with a current workload, will not allow one or more currently assigned compute clusters to meet the performance metric; The non-transitory readable storage medium of claim 12 further comprising:

14. The operation is determining that the workload can be serviced by one compute cluster less than the plurality of compute clusters; The non-transitory readable storage medium of claim 12 further comprising:

15. The operation is determining whether the current workload can be serviced by one compute cluster less than the plurality of compute clusters further comprises determining whether historical workload for a period leading up to the current time could have been serviced by one compute cluster less than the plurality of compute clusters while satisfying the performance metric; and deactivating the second compute cluster includes deactivating in response to determining that the historical workload for the time period was serviceable by one less compute cluster than the plurality of compute clusters; or providing queries to the virtual warehouse to each of the plurality of compute clusters includes routing queries based on the session from which the queries are submitted; The non-transitory readable storage medium of claim 14 further comprising:

Images