Workflow-based scheduling and batch processing in multi-tenant distributed systems

Abstract

Operation requests received from tenants are added to a queue that varies from tenant to tenant. Tenant scheduled work items are added to an execution queue shared with other tenants. When a tenant scheduled work item is executed, it copies up to a defined number of scheduled operations from a queue that varies from tenant to the execution queue. The tenant scheduled work item then re-adds itself to the execution queue. While the operation is being executed and before the tenant scheduled work item is executed again, other tenants have the opportunity to queue their own operations. The tenant scheduled work item selects scheduled operations from the queue that varies from tenant to tenant in the order in which the scheduled operations were originally requested until one of several conditions is met. The conditions can be based on how many operations are in progress, what kind of operations are in progress, and / or dependencies between different types of operations.

Description

Background Art

[0001] Scalability and fairness of computing resource allocation are ongoing technical challenges for multi-tenant distributed applications. Traditional distributed applications are single-tenant, i.e., each deployment serves a single customer, and therefore fairness is not an issue. Moreover, although scalability is a concern, single-tenant applications have more predictable and stable resource requirements due to their single-customer nature.

[0002] Multi-tenant applications (where each deployment serves multiple customers) are prone to over-allocating resources to one tenant and starving the remaining tenants of resources. This lack of fairness results in a poor user experience for the tenants that are starved of resources. At the same time, multi-tenant applications tend to have higher workloads than single-tenant applications, and therefore scaling is often a greater challenge.

[0003] Multi-tenant applications are typically deployed to server farms—a collection of compute nodes, storage, network bandwidth, and other computing resources. To efficiently utilize the compute nodes, applications can queue work items for the compute nodes to consume. When a compute node completes a work item, it requests another work item from the queue. In this way, the compute nodes can be kept busy, thereby improving application efficiency.

[0004] However, some tenants may request thousands or even millions of operations in a short period of time. Queuing these operations to be processed sequentially may delay the remaining tenants for minutes or even hours at a time. This unfairness results in a poor customer experience and may have financial consequences depending on whether a service level agreement is in place. Starving the remaining tenants of resources may also cause the server farm to perform less efficiently, because the users of the delayed tenants may retry their delayed operations multiple times, consuming additional bandwidth and processing power. In severe cases, retries may consume so much bandwidth and processing power that the distributed application crashes and needs to be restarted.

[0005] One challenge in scaling multi-tenant applications is to efficiently and correctly handle operations with dependencies. For example, if an application-specific operation (e.g., renaming a mailbox in an email application) is executed while a tenant-level operation (e.g., renaming the tenant associated with the email application) is in progress, it may cause data corruption. At the same time, while processing each operation from a given tenant in sequence can avoid correctness issues, performance and therefore scalability will be impaired.

[0006] The present disclosure made herein is proposed with respect to these and other considerations. The subject matter claimed herein is not limited to embodiments that solve any disadvantages or operate only in environments such as those described above. Instead, this background is provided only to illustrate an exemplary technical field in which some embodiments described herein can be practiced. Summary of the invention

[0007] One goal of the disclosed configuration is to load balance operations requested by users. Load balancing can be achieved in part by batching the requested operations. By democratically executing batch operations from different tenants, resource utilization fairness is improved.

[0008] In some configurations, operation requests received from tenants are added to the execution queue as work items. Work items can be added in batches. After one or more computing nodes have processed a batch, another batch can be added to the execution queue. By not queuing the next batch until the current batch is processed, other tenants have time to add their operations to the execution queue, thereby promoting fair access to computing resources. Batches are selected in a way that maximizes throughput without risking data corruption.

[0009] For example, a tenant administrator might want to rename all web pages hosted in a collaborative application. To do this, the tenant administrator can send a rename operation request for each web page. For a large tenant, this could amount to millions of rename requests. If these operations were added to the application's execution queue as they arrived, all farm resources could be consumed to perform the operation for a single tenant for an unacceptably long period of time. As millions of operations are processed, other tenants will be starved of resources.

[0010] To address this concern, in one configuration, operations are first added as work items to a tenant-specific work item queue (hereinafter referred to as a "tenant queue"). Operations that have been added to a tenant queue are considered scheduled because the operation is scheduled to be executed at a later time. Operations wait in the tenant queue until they are moved to the execution queue in batches for processing. In some configurations, the tenant queue and the execution queue are stored in the same database table (using an "OperationType" column to distinguish which queue a given work item belongs to).

[0011] Scheduled operations can be moved from a tenant queue to an execution queue through a novel work item - a tenant scheduling work item (hereinafter referred to as a "tenant scheduler"). A tenant scheduler can be placed in an execution queue along with operations and other work items. When executed by a compute node, the tenant scheduler selects a batch of scheduled operations from the tenant's queue and adds them to the execution queue. The tenant scheduler then re-adds itself to the execution queue, causing itself to be executed again after all operations in the batch have started. In this way, the tenant scheduler iteratively adds batches of operations to the execution queue. During the intervention time between iterations of the tenant scheduler, other tenants have the opportunity to queue their own operations, allowing fair access to server farm resources.

[0012] In some configurations, the tenant scheduler can batch operations in the order in which they were added to the tenant queue, i.e., without reordering or skipping operations. The tenant scheduler can add operations to a batch until one of several conditions is met. Once one of the conditions is met, the tenant scheduler can move the batch of operations to the execution queue.

[0013] In some configurations, a condition is satisfied when the number of operations in the execution queue plus the number of operations already in the batch is greater than or equal to a defined number. A similar condition is satisfied when the number of operations in the execution queue plus the number of operations in progress plus the number of operations already in the batch is greater than or equal to a defined number. The defined number may be calibrated to balance the efficiency of batching a larger number of operations with the disadvantage of starving other tenants of resources. One value of the defined number used in practice is 100, although any other value is similarly contemplated.

[0014] In some configurations, a condition is met if an operation, if executed in parallel with an operation already in progress, queued, or added to a batch, could result in data corruption. The tenant scheduler may detect the possibility of data corruption by comparing the type of operation at the head of the tenant queue with the type of operation in progress, in the execution queue, or added to a batch. For example, if an application-specific operation (such as renaming a folder in an email application) is executed while a tenant-level operation (e.g., tenant rename, tenant move, etc.) is in progress, it may result in data corruption. Thus, if the tenant scheduler encounters an application-specific operation while a tenant-level operation is in progress, in the execution queue, or added to a batch, the tenant scheduler may stop adding operations to the batch.

[0015] In some configurations, a condition is met whenever a tenant-level operation is in progress - that is, a tenant-level operation is being executed by a compute node. By not batching any operations while a tenant-level operation is in progress, tenant-level operations are serialized, that is, an ongoing tenant-level operation is allowed to complete before another tenant-level operation can be added to the execution queue. Similarly, application-specific operations are not added to the execution queue until an ongoing tenant-level operation completes. This prevents data corruption because application-specific operations may be invalidated by ongoing tenant-level operations.

[0016] Additionally or alternatively, another condition is met when there is an existing tenant-level operation in the execution queue and there are one or more application-specific operations in progress. In this scenario, the in-progress application-specific operation will be allowed to complete, after which the tenant-level operation will execute. According to the "do nothing while a tenant-level operation is in progress" rule described above, the tenant-level operation will then be fully executed before another scheduled operation of any type is added to the execution queue.

[0017] By complying with some or all of the conditions listed above, the tenant scheduler can balance throughput requirements with data integrity requirements. Specifically, operations that can be safely executed in parallel are allowed to do so, while potentially damaging operations are executed serially. Thus, in addition to ensuring fairness across tenants, batching operations for execution as described herein improves the scalability of the application. Technical benefits other than those described herein may also be achieved according to implementations of the disclosed technology.

[0018] It should be understood that the above-described subject matter may also be implemented as a computer-controlled device, a computer process, a computing system, or an article of manufacture such as a computer-readable medium. These and various other features will become apparent by reading the following detailed description and reviewing the associated drawings. This summary is provided to introduce a selection of concepts that are further described below in the detailed description in a simplified form.

[0019] This summary is neither intended to identify key features or essential features of the claimed subject matter, nor is this summary intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The detailed description is made with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items of a plurality of items may use reference numbers with letters of the letter sequence to refer to each individual item. General references to items may use specific reference numbers without the letter sequence.

[0021] Figure 1 is a block diagram illustrating a distributed collaborative application including a workflow scheduler as disclosed herein.

[0022] Figure 2 is a block diagram illustrating an example tenant list associated with a distributed collaboration application.

[0023] Figure 3Ais an illustration of a web page generated by a collaboration server before a site rename operation and / or a tenant rename operation has occurred.

[0024] Figure 3B is an illustration of a web page rendered by a collaboration server after a site rename operation and / or a tenant rename operation has occurred.

[0025] Figure 4 is a block diagram illustrating an example of a workflow scheduler queuing operations received from multiple tenants.

[0026] Figure 5 is a block diagram illustrating work item execution queues stored in the same work item queue and tenant-specific work item queues.

[0027] Fig. 6A is a block diagram illustrating that a workflow scheduler adds requested operations to a queue that varies by tenant.

[0028] Figure 6B is a block diagram illustrating a workflow scheduler adding a tenant scheduler work item to an execution queue.

[0029] Fig. 7A is a block diagram illustrating that a compute node retrieves and executes a tenant scheduler.

[0030] Figure 7B is a block diagram illustrating a compute node executing a tenant scheduler to copy scheduled operations from a tenant queue to an execution queue.

[0031] Figure 7C is a block diagram illustrating a compute node executing a tenant scheduler to re-add itself to an execution queue.

[0032] Fig.7D is a block diagram illustrating a compute node executing queued operations.

[0033] Fig. 8A is a block diagram illustrating that a workflow scheduler adds a requested operation to another tenant-specific work item queue if a compute node processes the queued operation.

[0034] Figure 8B is a block diagram illustrating that a workflow scheduler adds a tenant scheduler to an execution queue while a compute node processes queued operations.

[0035] Figure 8C is a block diagram illustrating a compute node executing a tenant scheduler retrieved from an execution queue after a previous batch of operations has completed execution.

[0036] Fig.8D is a block diagram illustrating a compute node executing a tenant scheduler to copy scheduled operations from a tenant queue to an execution queue.

[0037] Fig. 8E is a block diagram illustrating a compute node executing a tenant scheduler to requeue itself into an execution queue.

[0038] Figure 8F is a block diagram illustrating a compute node executing another tenant scheduler from an execution queue.

[0039] Figure 8G is a block diagram illustrating a compute node executing another tenant scheduler to add a scheduled operation to an execution queue.

[0040] Figure 8H is a block diagram illustrating a compute node executing another tenant scheduler to add itself back to the execution queue.

[0041] Fig. 9 is a flow chart illustrating an example operating procedure in accordance with the described implementations.

[0042] Fig.10 is a flow chart illustrating an example operating procedure in accordance with the described implementations.

[0043] Fig.11 is a flow chart illustrating an example operating procedure in accordance with the described implementations.

[0044] Fig.12 is a computer architecture diagram illustrating a computing device architecture for a computing device capable of implementing various aspects of the techniques and technologies presented herein.

[0045] Detailed Description

[0046] The technology disclosed herein provides improvements in the implementation of workflow-based scheduling and batch processing in multi-tenant distributed applications. In some embodiments, distributed applications are implemented in data centers, server farms, or other networked environments (including those that provide virtualized services). The disclosed technology can improve the fairness of access to computing resources. The disclosed technology can also increase performance and scalability by increasing parallelization in a manner that does not cause data corruption. The disclosed technology can also provide other technical benefits that are different from those described herein.

[0047] Figure 1 is a block diagram 100 illustrating a distributed collaboration application 106 including a workflow scheduler 108. The distributed collaboration application 106 is depicted as executing on a server farm 102A located within a computing cloud 104A. In addition to the workflow scheduler 108, the distributed collaboration application 106 may also include one or more tenants 110 and a work item queue 112.

[0048] Figure 1Multiple computing clouds 104 are illustrated hosting multiple server farms 104 around the world. A distributed collaboration application 106 is depicted as being deployed to server farm 104A, but the distributed collaboration application 106 can collaborate with other deployments of the distributed collaboration application 106 in other server farms 102. The distributed collaboration application 106 is distributed because it executes across multiple computing devices in the server farms 102 - collaboration with other deployments in other server farms is an additional layer of distribution, but is not required to implement some of the techniques disclosed herein.

[0049] Figure 1 Also illustrated are users 114, each of which is associated with a particular deployment of the distributed collaboration application 106. While there are no physical limitations preventing users anywhere in the world from being associated with a deployment in any server farm 102, users are often associated with tenants in the same geographic region as the server farm 102. Such an arrangement may be made to meet regulatory requirements in different jurisdictions, for administrative convenience, etc.

[0050] In some configurations, the distributed collaboration application 106 is used to share documents and files, host websites, and implement the legal, information management and procedural requirements of the organization in which it is deployed. Examples of collaboration applications include Microsoft SharePoint TM , Google Drive TM , Atlassian Confluence TM 、Box for Business TM and Drupal TM However, other types of distributed applications are similarly contemplated, including distributed email servers, web servers, application servers, or any other type of application that can operate in a distributed environment.

[0051] In some configurations, the distributed collaborative application 106 is multi-tenant because a single deployment serves multiple tenants. This is in contrast to a single-tenant deployment in which a single deployment serves a single tenant. Multi-tenant deployments share resources between tenants that are typically in the same computing cloud 104 or server farm 102, while single-tenant deployments use dedicated resources for one tenant and are more likely to be installed in an on-premise location. Multi-tenant deployments have several advantages, including better resource utilization due to resource sharing. In contrast, single-tenant deployments are typically over-provisioned to account for rare periods of high demand, otherwise wasted capacity. Other benefits of multi-tenant deployments include reduced administrative overhead, reduced electricity costs, and professionally managed security.

[0052] In some configurations, a tenant refers to a company, a government or non-profit entity, or any other collection of individuals that have sought to use the distributed collaboration application 106. Each tenant of the distributed collaboration server 106 can be associated with a name (e.g., "NegaCorp"). The tenant name can be used to identify the tenant for administrative functions. The tenant name can also be incorporated into the tenant's address (such as a uniform resource locator (URL)) and any items stored in the tenant.

[0053] Figure 2 2 is a block diagram 200 illustrating an example list of tenants 110 associated with a distributed collaboration application 106. In some configurations, each tenant 202 represents a different entity, such as "NegaCorp," "PosiIndustries," "AntiGroup," and "SpinPartners," that uses the distributed collaboration application 106. Although four tenants are listed, this is for illustration purposes only—a single deployment of the distributed collaboration application 106 may support hundreds of thousands or millions of tenants.

[0054] In some configurations, the collaboration application allows tenants to establish multiple sites 204. Each site may represent a self-contained unit of functionality available to teams within the tenant organization. For example, "NegaCorp" has site 204A1 "HumanResources", site 204A2 "Sales", and site 204A3 "Research". Each site may operate independently of the other sites. For example, documents, files, lists, websites, and other items stored for the "HumanResources" site 204A1 may be logically or physically independent of documents, files, lists, websites, and other items stored for the "Sales" site 204A2.

[0055] In some configurations, each site stores items in a different database. In another configuration, items from different sites within a tenant are stored in the same database but sharded into different database partitions based on the site name. In some configurations, site operations are an example of application-specific operations that are distinct from tenant-level operations, such as those described below in conjunction with Figure 3B discussed.

[0056] Figure 3A 300 is a diagram of a web page 302 generated by the collaboration server 106 before a site rename operation and / or a tenant rename operation has occurred. URL 304A is an example of a URL that a user can take to navigate to a specific site of a specific tenant. In this example, the site is named "HumanResources" and the tenant is named "NegaCorp".

[0057] Figure 3B3 is an illustration 300 of a web page 302 generated by a collaboration server 106 after a site rename operation and / or a tenant rename operation has occurred. Specifically, the tenant name "NegaCorp" has been replaced with the name "OmniCorp," and the site name "HumanResources" has been replaced with "HRWeb." These operations may be performed independently of one another, e.g., a tenant rename operation would have been performed without a site rename operation, and vice versa.

[0058] The distributed collaboration application 106 may support operations other than tenant rename and site rename. Some operations may be administrative, such as provisioning tenants, provisioning sites, deleting tenants, deleting sites, adding / updating users, etc. Other operations may reflect user operations, such as renaming files, moving files, updating contact lists, editing documents, etc.

[0059] As briefly discussed above, operations can also be categorized as tenant-level or application-specific. Tenant-level operations are related to the tenant and its operations. Tenant-level operations are generally similar, if not identical, across different types of applications. For example, tenant provisioning, tenant renaming, and tenant moving all apply equally to distributed collaboration applications as they do to distributed email applications, distributed source control server applications, and so on. Application-specific operations, on the other hand, are specific to the type of application. An email server application may support folder renaming, calendar operations, and other email-specific operations, while a collaboration server may allow sites to be renamed or moved, or allow for files, documents, lists, and other items to be created, deleted, renamed, and so on.

[0060] In some configurations, the distributed collaboration application 106 supports multiple sites within a given tenant. Users may have access to different sites within a tenant, but files, documents, lists, and other items from different sites may be stored in separate repositories. For example, each site provisioned by a tenant may have a distinct database, data store, or other mechanism for storing data. In other configurations, each site provisioned by a tenant may use a single database, but shard items based on site name—that is, store items from each site in distinct physical storage locations.

[0061] Figure 44 is a block diagram 400 illustrating an example of a workflow scheduler 108 queuing operations 404 received from multiple tenants 202. In some configurations, a user of a tenant 202 sends an operation 404 to the workflow scheduler 108. A single user (e.g., an administrator) may send many operations at once. For example, a tenant administrator may enforce a document retention policy by deleting hundreds, thousands, or millions of documents within a tenant (or within a site that is part of a tenant) with a single command. In other cases, many tenant users operating concurrently may each send an operation. Using the techniques described in conjunction with FIGS. 6-8, the workflow scheduler 108 divides the operation requests from different tenants into batches, thereby interleaving them into the work item queue 112 for processing.

[0062] In some configurations, the work item queue 112 processes any type of work item supported by the distributed collaborative application 106. The most straightforward example of a work item is a requested operation, such as tenant renaming, document movement, granting user access rights, etc. In some cases, the requested operation can be implemented using multiple work items. For example, an operation for migrating a user to a different geographic area may involve freezing the user's account, restoring the user's account in the destination area, and finally deleting the user's account in the originating area. Each of these actions can be performed by a different work item, with the freezing and deletion actions being performed by the local distributed collaborative application 106, and the recovery action being performed by the remote distributed collaborative application. The operation implemented using multiple work items may include the conditions of each work item that the tenant scheduler will check before adding the work item to the batch.

[0063] Another type of work item is a novel tenant-scheduled work item, such as tenant scheduler 408. Tenant scheduler 408 is created by workflow scheduler 108 to achieve fairness and improved performance between tenants, as discussed herein. For example, instead of adding operation 404B directly to work item queue 112 for execution by a compute node, workflow scheduler 108 has queued tenant scheduler 408B into work item queue 112. When executed by a compute node, workflow scheduler 108 may select a batch of scheduled operations 406B to be added to work item queue 112. In this way, instead of adding thousands or even millions of operations to work item queue 112 when they are received from a particular tenant, thereby increasing the possibility of starving other tenants of resources, tenant scheduler 408 iteratively adds a batch of operations to work item queue 112, and then adds itself to work item queue 112 to be executed again at a later time.

[0064] Figure 55 is a block diagram 500 illustrating a work item execution queue 502 and a tenant-specific work item queue 504B stored in the same work item queue 112. In some configurations, each tenant is assigned a tenant-specific work item queue 504, e.g., tenant 202A corresponds to tenant-specific work item queue 504A, tenant 202B corresponds to tenant-specific work item queue 504B, and so on.

[0065] In some configurations, the work item queue 112 is stored as a database table. Each entry in the work item database table may have an operation type column (e.g., "OperationType") that may be used to determine whether the work item is part of a work item execution queue 502 (hereinafter referred to as "execution queue" 502) or one of the tenant-specific work item queues 504 (hereinafter referred to as "tenant-specific queues" 504). For example, if the work item is an operation queued in a tenant-specific queue 504, the value of the "OperationType" column may be "TenantQueue". Conversely, if the OperationType is "ExecutionQueue", the work item is currently in the execution queue 502.

[0066] In some configurations, the database table underlying the work item queue 112 includes a parent ID column (e.g., 'ParentId') that indicates which tenant the work item is associated with. In some cases, the 'ParentId' column refers to the tenant scheduler that has, is, or will schedule the work item for execution. When the value of the 'OperationType' column is 'TenantQueue', the 'ParentId' column can be used to determine which tenant-specific queue 504 the work item is associated with.

[0067] Fig. 6A 6 is a block diagram 600 of the workflow scheduler 108 illustrating adding a requested operation 404A to a tenant-specific queue 604A. The requested operation 404A may be received from one or more users of the tenant 202A. The requested operations 404A may arrive all in a single request, one operation per request, or more than one operation per request. As depicted, the workflow scheduler 108 has received 1,000,003 requested operations and has enqueued them all into corresponding tenant-specific queues 604A.

[0068] Figure 6B is a block diagram 600 illustrating the workflow scheduler 108 adding a tenant scheduler 608A1 to the execution queue 602 . Figure 6BTenant scheduler 608A1 is described as "NegaCorp tenant scheduler 608A1". Figure 6B 608A1 and tenant 202A—the name “NegaCorp” is not necessarily stored in the execution queue 602 or the underlying work item table.

[0069] Fig. 7A 7 and 8 continue the same example from FIG. 6. In some configurations, the computing node 702 is a computing device within the server farm 102A that retrieves and executes the work item at the head of the execution queue 602. In some configurations, the computing node 702 is a virtual machine, while in other cases, the computing node 702 is a real physical machine. The computing node 702 can iteratively check the execution queue 602 to obtain an available work item and execute it if possible. In this way, the computing node 702 can execute operations requested by tenants, tenant schedulers, and other work item types.

[0070] Figure 7B 7 is a block diagram 700 illustrating that a compute node 702B executes a tenant scheduler 108 to copy scheduled operations 406A from a tenant queue 604A to an execution queue 602. In some configurations, up to a defined number of scheduled operations 406A are retrieved and added as a batch to the execution queue 602. As depicted, the tenant scheduler 608A1 retrieves 100 scheduled operations from the tenant-specific queue 604A and copies them to the execution queue 602 as a batch of operations 710A1 (x100).

[0071] Figure 7C is a block diagram 700 illustrating the compute node 702B executing the tenant scheduler 608A1 to re-add itself to the execution queue 602. In some configurations, the tenant scheduler 608A1 makes a copy of itself as the tenant scheduler 608A2 in the execution queue 602. In other configurations, the tenant scheduler 608A1 creates a new instance of the tenant scheduler 608A associated with the tenant 202A.

[0072] In some configurations, the tenant scheduler 608A2 will be executed a second time by re-adding the tenant scheduler 608A2 to the execution queue 602. When it is executed, the tenant scheduler 608A2 adds additional scheduled operations 406A to the execution queue 602 and may again re-add itself to the execution queue 602 if the tenant 202A has remaining unprocessed operations. In this way, the scheduled operations 406A are iteratively added to the execution queue 602, thereby allowing other tenants to access the execution queue 602 without starving for resources.

[0073] Fig.7D 7 is a block diagram 700 illustrating a computing node 702 executing a queued operation 404A. As depicted, the queued operation 404A being executed is part of a batch of queued operations 710A. As depicted, each of computing nodes 702A, 702B, and 702C has downloaded and is executing one of queued operations 404A1, 404A2, and 404A3. 97 of the original 100 queued operations remain in the batch of operations 710A.

[0074] Fig. 8A 8 is a block diagram 800 illustrating that the workflow scheduler 108 adds a requested operation 404B to a tenant-specific work item queue 406B while a queued operation 404A is processed by a compute node 702. In some configurations, similar to the receipt of the operation 404A, the operation 404B may be received from a single user performing 500,000 operations, received by 500,000 users each performing one operation, or received by some fewer number of users each requesting at least one operation.

[0075] In some configurations, operations 404B are stored in a tenant-specific work item queue 604B (hereinafter referred to as a “tenant-specific queue” 406B). Like tenant-specific queue 604A, tenant-specific queue 604B can be stored in the same physical database table as execution queue 602.

[0076] In some configurations, while the requested operation 404B is being queued in the tenant-specific queue 604B, the compute node 702 continues to process queued operations 404A from the batch of operations 710A1. At the time of illustration, 65 queued operations 404A remain in the batch of operations 710A1.

[0077] Figure 8B is a block diagram 800 illustrating the workflow scheduler 108 adding a tenant scheduler 608B1 to the execution queue 602 if a compute node 702 processes a queued operation 404A. In some configurations, the workflow scheduler 108 creates the tenant scheduler 608B1 to schedule the execution of the operation 404B.

[0078] In some configurations, while the workflow scheduler is creating the tenant scheduler 608B1 and adding it to the execution queue 602, the compute node 702 continues to process queued operations 404A from a batch of operations 710A1. At the time of illustration, 42 queued operations 404A remain in the batch of operations 710A1.

[0079] Figure 8C is a block diagram 800 illustrating the execution of a tenant scheduler 608A2 retrieved from the execution queue 602 after a batch of operations 710A have completed execution by a compute node 702A.

[0080] Fig.8D 8 is a block diagram 800 illustrating that a compute node 702A executes a tenant scheduler 608A2 to copy scheduled operations 406A from a tenant queue 604A to an execution queue 602. In some configurations, up to a defined number of scheduled operations 406A are copied from a tenant queue 604A to an execution queue 602. Fig.8D As illustrated in , the defined number may be 100, and thus 100 scheduled operations 406A are selected as a batch of operations 710A2. The tenant scheduler 608A2 adds the batch of operations 710A2 to the execution queue 602.

[0081] In some configurations, if some queued operations 404A remain in the execution queue 602, the number of scheduled operations 406A selected for the batch of operations 710A2 will be reduced so that the total number of active queued operations 404 is less than or equal to the defined number. In some configurations, the active operations 404 are either in progress (i.e., being executed by one of the compute nodes 702) or queued in the execution queue 602.

[0082] In some configurations, queued operations 404A may remain active from the last batch of operations 710A1 because they take a long time to complete execution. For example, if one of the operations 404A from a batch of queued operations 710A1 takes longer to complete execution than the time it takes for all operations in the batch of operations 710A1 to begin execution and for the tenant scheduler 608A2 to begin selecting the next batch, the tenant scheduler 608A2 may select 99 scheduled operations 406 for the next batch.

[0083] Fig. 8E is a block diagram 800 illustrating that the compute node 702A executes the tenant scheduler 608A2 to requeue itself into the execution queue 602. Figure 7C As discussed, the tenant scheduler 608A2 may copy itself into the execution queue 602 as the tenant scheduler 608A3, or the tenant scheduler 608A2 may create a similar tenant scheduler 608A3 and add it to the execution queue 602. By re-adding itself to the execution queue 602, the tenant scheduler 608A iteratively adds the scheduled operations 406A in batches to the execution queue 602.

[0084] Figure 8Fis a block diagram 800 illustrating compute node 702C executing another tenant scheduler 608B1 retrieved from execution queue 602. By executing tenant scheduler 608B1, operation 404B requested by tenant 202B can be queued for execution without first executing all 1,000,003 operations received from tenant 202A.

[0085] Figure 8G 8 is a block diagram 800 illustrating that a compute node 702C executes a tenant scheduler 608B1 to add a scheduled operation 406B to an execution queue 602. In some configurations, the scheduled operation 406B is added to the execution queue 602 as a batch of operations 710B1 that includes operation 404B. When the tenant scheduler 608B1 adds the scheduled operation 406B to the execution queue 602, other compute nodes 702 are beginning to retrieve and process the queued operation 404A at the top of the execution queue 602.

[0086] Figure 8H is a block diagram 800 illustrating compute node 702C executing tenant scheduler 608B1 to re-add itself to execution queue 602. In some configurations, at least compute nodes 702A and 702B continue to execute queued operations 404A.

[0087] refer to Fig. 9 , operation 900 begins the procedure. Operation 900 may be implemented by a workflow scheduler 108 executing on one of the compute nodes 702 of the server farm 102. Operation 900 may be followed by operation 902. Operation 902 illustrates receiving a request from a tenant to perform an operation. Operation 902 may be followed by operation 904. Operation 904 illustrates adding the received operation to a tenant-specific work item queue.

[0088] Operation 904 may be followed by operation 906. Operation 906 illustrates determining whether a tenant scheduled work item exists for the tenant. If the answer is yes, operation 906 is followed by operation 910, where operation 900 ends. If the answer is no, operation 906 may be followed by operation 908. Operation 908 illustrates creating a tenant scheduled work item for the tenant and adding it to the execution queue 602. Operation 908 may be followed by operation 910, where operation 900 ends.

[0089] refer to Fig.10 , illustrating an example operating procedure according to the present disclosure. Fig.10 , operation 1000 begins the procedure. Operation 1000 may be performed by a tenant scheduled work item 608 executing on a compute node 702. Operation 1000 may be followed by operation 1002. Operation 1002 illustrates retrieving and executing a tenant scheduled work item from the head of the execution queue 602.

[0090] Operation 1002 may be followed by operation 1004. Operation 1004 illustrates determining whether any operations scheduled in the tenant-specific queue 604 can be added to the execution queue 602. If the answer is yes, operation 1004 may be followed by operation 1006. If the answer is no, operation 1004 may be followed by operation 1010. Operation 1006 illustrates copying up to a defined number of scheduled operations from the tenant-specific queue 604 to the execution queue 602.

[0091] Operation 1008 may be followed by operation 1008. Operation 1008 illustrates re-adding the tenant scheduled work item 406 to the execution queue 602. Operation 1008 may be followed by operation 1014, where operation 1000 ends. Operation 1010 illustrates determining whether any operations received from the tenant are still being executed. If the answer is yes, operation 1010 may be followed by operation 1008. If the answer is no, operation 1010 may be followed by operation 1012.

[0092] Operation 1012 illustrates marking the tenant scheduled work item as complete because all operations received from the tenant have completed execution. If the tenant scheduled work item is complete, it can be deleted, released, or allowed to be garbage collected. Operation 1012 can be followed by operation 1014, where operation 1000 ends.

[0093] refer to Fig.11 , illustrating an example operating procedure according to the present disclosure. Fig.11 , operation 1100 begins the procedure. Operation 1100 may be performed by a tenant scheduling work item 608 executing on a compute node 702. In some configurations, operation 1100 performs decision block 1004.

[0094] Operation 1100 may be followed by operation 1102. Operation 1102 illustrates determining whether the number of queued operations (and / or in-progress operations) is greater than the defined number of operations. If so, operation 1102 may be followed by operation 1110. If not, operation 1102 may be followed by operation 1104.

[0095] Operation 1104 illustrates determining whether a tenant-level operation for the tenant is already in progress. If so, operation 1104 may proceed to operation 1106. If not, operation 1104 may proceed to operation 1100. Operation 1106 illustrates determining whether the execution queue 602 includes queued tenant-level operations and whether one or more site-level operations are in progress. If the answer is no, operation 1106 may proceed to operation 1108. If the answer is yes, operation 1106 may proceed to operation 1100.

[0096] Operation 1108 illustrates adding up to a defined number of scheduled operations from the tenant-specific queue 604 to the execution queue 602 in chronological order—that is, the scheduled operations are added to the execution queue 602 in the order in which they are received. Operation 1108 may proceed to operation 1112, where the process 1100 ends. Operation 1110 illustrates doing nothing—that is, ignoring any requests. By doing nothing, operation 1110 allows operations already in the execution queue 602 to be processed.

[0097] It should also be understood that the methods described may be terminated at any time and need not be fully executed. Some or all of the operations of these methods and / or substantially equivalent operations may be performed by executing computer-readable instructions included on a computer storage medium, as defined below. The term "computer-readable instructions" and variations thereof, as used in the specification and claims, are used herein to broadly include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0098] It should therefore be understood that the logical operations described herein are implemented as: (1) a series of computer-implemented actions or program modules running on a computing system; and / or (2) interconnected machine logic circuits or circuit modules within a computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are variously referred to as states, operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0099] For example, the operation of the described method is described herein as being implemented at least in part by a system component, which may include an application, a component, and / or a circuit. In some configurations, the system component includes a dynamic link library (DLL), a static link library, functionality implemented by an application programming interface (API), a compiled program, an interpreted program, a script, or any other executable instruction set. Data may be stored in one or more memory components in a data structure. Data may be retrieved from a data structure by addressing a link or reference to the data structure.

[0100] Although the following explanation involves Figure 1-11The components of the present invention are described herein, but it can be appreciated that the operations of the described methods can also be implemented in many other ways. For example, the methods can be implemented at least in part by a processor of another remote computer or local circuit. In addition, one or more operations of the methods can be alternatively or additionally implemented at least in part by a chipset working alone or in conjunction with other software modules. Any service, circuit or application suitable for providing the technology disclosed herein can be used in the operations described herein.

[0101] Fig.12 Additional details of an example computer architecture are shown that can implement various aspects of the embodiments described above. Fig.12 The computer architecture shown illustrates various aspects of a system, such as a conventional server computer, workstation, desktop computer, laptop computer, tablet, computing or processing system embedded in a device (such as a wearable device, automobile, home automation, etc.) or other computing device, and can be used to execute any software components presented herein. For example, Fig.12 The computer architecture shown may be used to execute any of the software components described above.

[0102] The computer architecture includes a baseboard 1202 or "motherboard," which is a printed circuit board to which a number of components or devices may be connected via a system bus or other electronic communication paths. In one illustrative embodiment, one or more central processing units ("CPUs") 1204 operate in conjunction with a chipset 1206. CPU 1204 may be a standard programmable processor that performs the arithmetic and logic operations required for the operation of the computing architecture.

[0103] The CPU 1204 performs operations by transitioning from one discrete physical state to the next by manipulating switching elements that distinguish between and change the states. Switching elements may generally include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide output states based on a logical combination of the states of one or more other switching elements, such as logic gates. These basic switching elements may be combined to create more complex logic circuits, including registers, adder-subtractors, arithmetic logic units, floating point units, and the like.

[0104] Chipset 1206 provides an interface between CPU 1204 and the rest of the components and devices on baseboard 1202. Chipset 1206 may provide an interface to RAM 1208 used as main memory in the computer architecture. Chipset 1206 may also provide an interface to computer-readable storage media such as read-only memory ("ROM") 1210 or non-volatile RAM ("NVRAM") for storing basic routines that help start the computing device and transfer information between various components and devices. ROM 1210 or NVRAM may also store other software components necessary for the operation of the computing architecture according to the embodiments described herein.

[0105] The computer architecture can operate in a networked environment using logical connections to remote computing devices and computer systems through a network 1214 such as a local area network. Chipset 1206 may include functionality for providing network connectivity through a network interface controller (NIC) 1288 such as a Gigabit Ethernet adapter. NIC 1288 is capable of connecting the computer architecture to other computing devices through network 1214. It should be understood that multiple NICs 1288 may be present in the computer architecture to connect the computer to other types of networks and remote computer systems. The network allows the computer architecture to communicate with remote services and servers (such as remote computers 1201). As can be appreciated, remote computers 1201 can be computing nodes 702 or any other computing device that is part of a server farm 102. In addition, as described above, remote computers 1201 can mirror and reflect data stored on computer architectures and host services that can provide data or processing for the technology described herein.

[0106] The computer architecture can be connected to a mass storage device 1226 that provides non-volatile storage for the computing device. The mass storage device 1226 can store system programs, applications, other program modules, and data that have been described in more detail herein. The mass storage device 1226 can be connected to the computer architecture through a storage controller 1215 connected to the chipset 1206. The mass storage device 1226 can be composed of one or more physical storage units. The storage controller 1215 can be connected to the physical storage unit through a serial attached SCSI ("SAS") interface, a serial advanced technology attachment ("SATA") interface, a fiber channel ("FC") interface, or other types of interfaces for physically connecting and transmitting data between a computer and a physical storage unit. It should also be understood that the mass storage device 1226, other storage media, and the storage controller 1215 may include a multimedia card (MMC) component, an eMMC component, a secure digital (SD) component, a PCI high-speed component, and the like.

[0107] The computer architecture can store data on the mass storage device 1226 by transforming the physical state of the physical storage unit to reflect the information being stored. In different implementations of the present specification, the specific transformation of the physical state may depend on various factors. Examples of these factors may include, but are not limited to: the technology used to implement the physical storage unit, whether the mass storage device 1226 is characterized as primary storage or secondary storage, etc.

[0108] For example, the computer architecture may store information to the mass storage device 1226 by issuing instructions via the storage controller 1215 to change the magnetic properties of a specific location within a disk drive unit, the reflective or refractive properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of the present specification, and the examples provided above are merely for ease of description. The computer architecture may further read information from the mass storage device 1226 by detecting the physical state or properties of one or more specific locations within the physical storage unit.

[0109] In addition to the mass storage device 1226 described above, the computer architecture may also access other computer-readable media to store and retrieve information, such as program modules, data structures, or other data. Although the operating system 1227, workflow scheduler 108, other data, and other modules are depicted as data and software stored in the mass storage device 1226, it should be understood that these components and / or other modules may be stored at least in part in other computer-readable storage media of the computer architecture. Although the description of the computer-readable media contained herein references mass storage devices such as solid-state drives, hard disks, or CD-ROM drives, it should be understood by those skilled in the art that computer-readable media may be any available computer storage media or communication media that can be accessed by the computer architecture.

[0110] Communication media includes computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any delivery medium. The term "modulated data signal" means a signal that has one or more of its characteristics changed or set in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Any combination of the above should also be included within the scope of computer readable media.

[0111] By way of example and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer architecture. For the purposes of this statement, the phrases "computer storage media", "computer readable storage media" and their variations do not include waves or signals themselves and / or communication media.

[0112] The mass storage device 1226 may store an operating system 1227 for controlling the operation of the computer architecture. According to one embodiment, the operating system includes a gaming operating system. According to another embodiment, the operating system includes a Microsoft Operating system. According to other embodiments, the operating system may include a UNIX, ANDROID, WINDOWS PHONE, or iOS operating system that may be obtained from its respective manufacturer. It should be understood that other operating systems may also be utilized. The mass storage device 1226 may store other system or application programs and data used by the computer architecture, such as any other software components and data described above. The mass storage device 1226 may also store other programs and data not specifically identified herein.

[0113] In one embodiment, the mass storage device 1226 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into the computer architecture, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. These computer-executable instructions transform the computer architecture by specifying how the CPU 1204 transitions between states as described above. According to one embodiment, the computer architecture can access a computer-readable storage medium storing computer-executable instructions that, when executed by the computer architecture, perform the operations described above with reference to Fig.12 The computing device may also include a computer-readable storage medium for performing any other computer-implemented operations described herein.

[0114] The computer architecture may also include one or more input / output controllers 1216 for receiving and processing input from a number of input devices such as a keyboard, mouse, microphone, headset, touch pad, touch screen, electronic stylus, image processing and gesture recognition devices, or any other type of input device. The input / output controller 1216 communicates with the input / output device 1225. The input / output controller 1216 may provide output to a display such as a computer monitor, flat panel display, digital projector, printer, plotter, or other type of output device. The input / output controller 1216 may provide input communication with other devices such as microphones, speakers, game controllers, and / or audio devices.

[0115] For example, the input / output controller 1216 may be an encoder and the input / output device 1225 may include a full speaker system with multiple speakers. The encoder may use spatialization techniques, and the encoder may process the audio output audio or output signal received from the application 1229. The encoder may utilize the selected spatialization techniques to generate a spatially encoded stream that is appropriately rendered to the input / output device 1225.

[0116] The disclosure presented herein may be considered in light of the following terms.

[0117] Example 1: A computer-implemented method for scheduling operations, the method comprising: receiving multiple requests to perform operations from one of multiple tenants of a distributed computing application; adding the multiple operations to a queue that varies depending on the tenant as multiple scheduled operations; adding a tenant scheduling work item to an execution queue, wherein the tenant scheduling work item causes one of multiple computing nodes to: create a batch of operations by selecting up to a defined number of scheduled operations from the multiple scheduled operations, adding the batch of operations to the execution queue, and re-adding the tenant scheduling work item to the execution queue; and wherein at least one of the multiple computing nodes executes the batch of operations.

[0118] Example 2: A computer-implemented method of Example 1, further comprising: receiving a second plurality of operations from a second tenant of the plurality of tenants; queuing the second plurality of operations in a second queue specific to the tenant as a second plurality of scheduled operations; and queuing the second tenant scheduling work item in the execution queue before the tenant scheduling work item adds all of the plurality of scheduled operations to the execution queue.

[0119] Example 3: The computer-implemented method of Example 1, wherein the execution queue and the tenant-specific queue are stored in the same database table.

[0120] Example 4: A computer-implemented method of Example 3, wherein scheduled operations stored in a tenant-specific queue, queued operations stored in an execution queue, and tenant-scheduled work items stored in an execution queue are represented by columns in the same database table, comprising: a site identifier column, a parent identifier column referring to the tenant scheduler, an operation type column referring to whether the operation is scheduled in a tenant-specific queue or in an execution queue, and a processing status column referring to whether the operation is scheduled, queued, started, or completed.

[0121] Example 5: The computer-implemented method of Example 4, further comprising: querying a database table to identify operations from one of the multiple tenants, grouped by an operation type column and a processing status column; and querying a database table to identify operations from one of the multiple tenants, grouped by a site identifier column.

[0122] Example 6: A computer-implemented method of Example 5, wherein a tenant scheduling work item selects up to a defined number of scheduled operations by adding scheduled operations to the batch until the number of queued operations in the execution queue plus the number of scheduled operations already added to the batch equals the defined number.

[0123] Example 7: The computer-implemented method of Example 5, wherein the plurality of operations include tenant-level operations that perform administrative operations on the tenant, and wherein the plurality of operations include application-specific operations that perform functionality specific to the distributed computing application.

[0124] Example 8: The computer-implemented method of Example 7, wherein the tenant scheduled work item stops adding scheduled operations to the batch when a tenant-level operation is being performed by one of the plurality of compute nodes.

[0125] Example 9: The computer-implemented method of Example 7, wherein when a tenant-level operation is queued in the execution queue and one or more application-specific operations are being executed by one or more of the plurality of compute nodes, the tenant scheduling work item stops adding scheduled operations to the batch.

[0126] Example 10: A computing device comprising: one or more processors; a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereon, which, when executed by the one or more processors, cause the computing device to perform operations including: receiving a first plurality of requests for operations from a first tenant of a plurality of tenants of a distributed computing application; adding the first plurality of operations to a first tenant-specific queue as a first plurality of scheduled operations; adding a first tenant scheduled work item to an execution queue, wherein the first tenant scheduled work item causes one of a plurality of computing nodes to: create a first batch of operations by selecting up to a defined number of scheduled operations from the first plurality of scheduled operations, adding the first batch of operations to the execution queue, and re-adding the first tenant scheduled work item to the execution queue; receiving a second plurality of operations from a second tenant of the plurality of tenants of the distributed computing application; queuing the second plurality of operations in a second tenant-specific queue as a second plurality of scheduled operations; and adding the second tenant scheduled work item to the execution queue before the first batch of operations has been executed.

[0127] Example 11: The computing device of Example 10, wherein the re-added first tenant schedules the work item to be executed by one of the plurality of computing nodes, thereby causing another batch of operations received from the first tenant to be added to the execution queue.

[0128] Example 12: The computing device of Example 10, wherein the scheduled operations selected for inclusion in the first batch of operations are selected in an order in which they are received by the distributed computing application.

[0129] Example 13: The computing device of Example 10, wherein the execution queue includes operations from the first tenant mixed with operations from the second tenant.

[0130] Example 14: The computing device of Example 10, wherein two or more of the plurality of computing nodes execute the second tenant scheduled work item and one or more of the first plurality of operations in parallel.

[0131] Example 15: A computer-readable medium having computer-executable instructions encoded thereon, which when executed cause one or more processing units of a computing device to perform a method comprising: receiving multiple requests for operations from one of multiple tenants of a distributed computing application; adding the multiple operations to a queue that is specific to the tenant as multiple scheduled operations; adding a tenant-scheduled work item to an execution queue, wherein the tenant-scheduled work item causes one of multiple computing nodes to: create a batch of operations by selecting from the multiple scheduled operations in the queue that is specific to the tenant until at least one of multiple conditions is met, add the batch of operations to the execution queue, and re-add the tenant-scheduled work item to the execution queue; and wherein at least one of the multiple computing nodes executes the batch of operations.

[0132] Example 16: The computer-readable medium of Example 15, wherein the batch of operations is created by selecting from the plurality of scheduled operations in an order in which the operations are received from one of the plurality of tenants.

[0133] Example 17: The computer readable medium of Example 16, wherein the operation can be associated with a delivery date and time, and wherein the operation is not added to the batch until the current date and time is at least the delivery date and time.

[0134] Example 18: The computer-readable medium of Example 17, wherein the re-added tenant scheduled work item is associated with a delivery date and time that is a defined amount of time after the tenant scheduled work item was re-added to the execution queue.

[0135] Example 19: The computer-readable medium of Example 15, wherein the scheduled operations include tenant-level operations and application-specific operations, and wherein the scheduled operations are selected for inclusion in the batch in the order in which they are received until: the tenant-level operation is being executed by one of the plurality of compute nodes, or

[0136] The tenant-level operation exists in the execution queue, and one or more application-specific operations are being executed by one or more of the plurality of compute nodes.

[0137] Example 20: The computer-readable medium of Example 15, wherein the operations include tenant-level operations and site-specific operations, and wherein executing the batch of operations includes executing any tenant-level operations serially, while operations from different sites may be executed in parallel.

[0138] Each of the processes, methods, and algorithms described in the above sections may be instantiated in whole or in part automatically in a code module executed by one or more computers or computer processors. The code module may be stored on any type of non-transient computer-readable medium or computer storage device, such as a hard disk, solid-state memory, and / or optical disk, etc. The processes and algorithms may be implemented in part or in whole with dedicated circuits. The results of the disclosed processes and process steps may be stored permanently or otherwise in any type of non-transient computer storage, such as, for example, volatile or non-volatile storage.

[0139] Each feature and process described above may be independently of each other, or combined in various ways. All possible combinations and sub-combinations are expected to fall within the scope of the present disclosure. In addition, in some implementations, certain method or process blocks may be omitted. The methods and processes described herein are also not limited to any particular order, and each block and the state associated therewith may be executed in other appropriate orders. For example, the described blocks or states may be executed in a different order than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in other forms. Blocks or states may be added or removed relative to the disclosed example embodiments. The example systems and components described herein may be configured differently from what is described. For example, relative to the disclosed example embodiments, each element may be added, removed, or rearranged.

[0140] It will also be understood that the various items are illustrated as being stored in memory or storage when being used, and for the purpose of memory management and data integrity, these items or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via cross-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least in part in the form of firmware and / or hardware, including but not limited to application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Therefore, the present invention may be implemented with other computer system configurations.

[0141] Unless otherwise specifically stated, or otherwise understood in the context as used, the conditional language used herein (such as "can", "can", "may" or "can", "for example", etc.) is generally intended to express that a particular embodiment includes and other embodiments do not include specific features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements and / or steps are required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without user input or prompts, whether to include or perform these features, elements and / or steps in any particular embodiment. The terms "include", "comprise", "have", etc. are synonyms and are used in an open form inclusively without excluding other elements, features, actions, operations, etc. In addition, the term "or" is used with its inclusive meaning (rather than with its exclusive meaning), so that when used, for example, when used to connect a list of elements, the term "or" represents one, some or all of the elements in the list.

[0142] Although certain example embodiments have been described, these embodiments are provided as examples only and are not intended to limit the scope of the invention disclosed herein. Therefore, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or irreplaceable. In fact, the new methods and systems described herein may be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention disclosed herein. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of some of the inventions disclosed herein.

Claims

1. A computer-implemented method for scheduling operations, the method comprising: receiving a plurality of requests to perform operations from one of a plurality of tenants of a distributed computing application; Add multiple operations to tenant-specific queues as multiple scheduled operations; A tenant-scheduled work item is added to an execution queue, wherein the tenant-scheduled work item causes one of the plurality of compute nodes to perform the following operations: creating a batch of operations by selecting up to a defined number of scheduled operations from the plurality of scheduled operations, adding the batch of operations to the execution queue, and Re-adding the tenant scheduled work item to the execution queue; and At least one of the plurality of computing nodes executes the batch of operations.

2. The computer-implemented method of claim 1 , further comprising: receiving a second plurality of operations from a second tenant of the plurality of tenants; enqueuing the second plurality of operations in a second tenant-specific queue as a second plurality of scheduled operations; Before the tenant scheduled work item adds all of the plurality of scheduled operations to the execution queue, a second tenant scheduled work item is queued in the execution queue.

3. The computer-implemented method of any one of claims 1 to 2, wherein: The execution queue and the tenant-specific queue are stored in the same database table.

4. The computer-implemented method of any one of claims 1 to 2, wherein: The scheduled operations stored in the tenant-specific queue, the queued operations stored in the execution queue, and the tenant-scheduled work items stored in the execution queue are represented by columns in the same database table, including: a site identifier column, a parent identifier column referring to the tenant scheduler, an operation type column indicating whether the operation is scheduled in the tenant-specific queue or in the execution queue, and a processing status column indicating whether the operation is scheduled, queued, started or completed.

5. The computer-implemented method of any one of claims 1-2, further comprising: querying the database table to identify operations from one of the plurality of tenants, grouped by the operation type column and the processing status column; as well as The database table is queried to identify operations from one of the plurality of tenants, grouped by the site identifier column.

6. The computer-implemented method of any one of claims 1-2, wherein: The tenant scheduling work item selects up to the defined number of scheduled operations by adding scheduled operations to the batch of operations until the number of queued operations in the execution queue plus the number of scheduled operations already added to the batch of operations equals the defined number.

7. The computer-implemented method of any one of claims 1-2, wherein: The plurality of operations include tenant-level operations that perform administrative operations on the tenant, and wherein the plurality of operations include performing application-specific operations that vary depending on functionality of the distributed computing application.

8. The computer-implemented method of any one of claims 1-2, wherein: When a tenant-level operation is being executed by one of the plurality of computing nodes, the tenant scheduling work item stops adding the scheduled operation to the batch operation.

9. The computer-implemented method of any one of claims 1-2, wherein: When a tenant-level operation is queued in the execution queue and one or more application-specific operations are being executed by one or more of the plurality of compute nodes, the tenant scheduling work item stops adding the scheduled operation to the batch of operations.

10. A computing device for scheduling operations, comprising: one or more processors; a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereon that, when executed by the one or more processors, cause the computing device to perform operations comprising: receiving a first plurality of requests for operations from a first tenant of a plurality of tenants of a distributed computing application; adding a first plurality of operations to a first tenant-specific queue as a first plurality of scheduled operations; A first tenant scheduling work item is added to an execution queue, wherein the first tenant scheduling work item causes one of the plurality of computing nodes to perform the following operations: creating a first batch of operations by selecting up to a defined number of scheduled operations from the first plurality of scheduled operations, adding the first batch of operations to the execution queue, and Re-adding the first tenant scheduled work item to the execution queue; receiving a second plurality of operations from a second tenant of the plurality of tenants of the distributed computing application; enqueuing the second plurality of operations in a second tenant-specific queue as a second plurality of scheduled operations; Before the first batch of operations has been executed, a second tenant scheduled work item is added to the execution queue.

11. A computer-readable medium having computer-executable instructions encoded thereon, the computer-executable instructions, when executed, causing one or more processing units of a computing device to perform a method comprising: receiving a plurality of requests for operations from one of a plurality of tenants of a distributed computing application; Add multiple operations to tenant-specific queues as multiple scheduled operations; A tenant-scheduled work item is added to an execution queue, wherein the tenant-scheduled work item causes one of the plurality of compute nodes to perform the following operations: creating a batch of operations by selecting from the plurality of scheduled operations in the tenant-specific queue until at least one of a plurality of conditions is satisfied, adding the batch of operations to the execution queue, and Re-adding the tenant scheduled work item to the execution queue; and At least one of the plurality of computing nodes executes the batch of operations.

12. The computer-readable medium of claim 11, wherein: The batch of operations is created by selecting from the plurality of scheduled operations in the order in which the operations are received from one of the plurality of tenants.

13. The computer-readable medium of any one of claims 11-12, wherein: The operation can be associated with a delivery date and time, and wherein the operation is not added to the batch of operations until the current date and time is at least the delivery date and time.

14. The computer-readable medium of any one of claims 11-12, wherein: The re-added tenant scheduled work item is associated with a due date and time, which is a defined amount of time after the tenant scheduled work item was re-added to the execution queue.

15. The computer-readable medium of any one of claims 11-12, wherein: The scheduled operations include tenant-level operations and application-specific operations, and wherein the scheduled operations are selected for inclusion in the batch of operations in the order in which they are received until: The tenant-level operation is being performed by one of the plurality of compute nodes, or Tenant-level operations are present in the execution queue, and one or more application-specific operations are being executed by one or more of the plurality of compute nodes.

16. A computer-readable storage medium having instructions, which, when executed, cause a machine to perform the method of any one of claims 1-2.

17. A computer system comprising means for executing the method according to any one of claims 1-2.

Images