Asymmetric collaborative queue management of messages

Abstract

Improved techniques for responding to a server submerged by a message are disclosed herein. For example, a message may be initially placed in an "incoming" queue, once the server accepts a corresponding message for processing, the message may be placed in the accepted queue, and an indication is generated indicating that the message will be processed. When the number of messages in the accepted queue meets or exceeds a prescribed threshold, new messages addressed to the incoming queue will be rejected. Further, computing resources are allocated to the incoming queue until messages in the incoming queue are processed (or other conditions are met). Once the incoming queue is empty or other conditions are met, resources are allocated to the accepted queue. Once the number of messages in the accepted queue meets or falls below a second threshold, new messages will be again allowed to enter the incoming queue.

Description

[0001] This application is a divisional application of the application with application number 201980040246.2 and invention title "Asymmetric Cooperative Queue Management of Messages" filed on June 7, 2019. Background Art

[0002] Message servers handle a large workload, jointly processing hundreds of billions of messages globally every day. However, when incoming messages exceed the server's capacity to process these messages, the message server may become overwhelmed. Messages sent to an overwhelmed server are typically retained in a queue until they are processed, which results in latency. When the latency is too long, the message may time out, causing the message sender to retry the message. Additionally, the overwhelmed message server may continue to receive new messages, increasing the latency and timeouts. Furthermore, as more and more resources are dedicated to scheduling messages, the resulting backlog reduces computer efficiency, thereby reducing the resources available for actual message processing. Thus, it can be understood that there is a continuous need to improve the technology for a server overwhelmed with messages to respond.

[0003] In view of these and other considerations, the disclosure herein is presented. Summary of the Invention

[0004] Improved techniques for a server overwhelmed with messages to respond are disclosed herein. Initially, messages can be placed in an "incoming" queue. Once the server receives a corresponding message for processing, the message can be placed in the accepted queue and an indication that the message will be processed is generated. When the number of messages in the accepted queue meets or exceeds a specified threshold, new messages addressed to the incoming queue are rejected. Additionally, computing resources are not allocated to the incoming queue until the messages in the incoming queue have been processed (or other conditions are met). Once the incoming queue is empty or other conditions are met, resources are allocated to the accepted queue. Once the number of messages in the accepted queue meets or drops below a second threshold, new messages are again allowed to enter the incoming queue.

[0005] In some embodiments, messages such as emails are placed in an "incoming" queue. The incoming queue contains messages not accepted by the server, and there is no guarantee that the incoming message will be processed. Once the server accepts a message, it can be removed from the incoming queue and placed in an "accepted" queue. It is guaranteed that the messages in the accepted queue will be processed by the server.

[0006] In some embodiments, the incoming queue and the accepted queue are collaborative - they share a common pool of computing resources (e.g., processors, memory, network bandwidth, etc.), and they cede these resources to each other based on various criteria. For example, in a normal operating state (i.e., when neither queue has exceeded a threshold number of messages), computing resources can be allocated proportionally to the size of each queue. However, in some embodiments, the accepted queue can be given more or slightly more than proportional resources to avoid accumulating accepted messages.

[0007] Additionally or alternatively, when the number of messages in the accepted queue meets or exceeds a threshold, the server can respond by blocking new messages from being added to the incoming queue. While the incoming queue is blocked, the server can still continue to allow messages to enter the accepted queue. Continuing to allow messages to enter the accepted queue is counterintuitive because the response is triggered by a large number of messages in the accepted queue. Rejecting messages from the incoming queue but allowing messages to enter the accepted queue can prevent a runaway situation where the number of received new messages exceeds the number that can be processed. At the same time, messages that already exist in the incoming queue can still be added to and processed by the accepted queue.

[0008] Meanwhile, the server can allocate some or all of the computing resources to the incoming queue. Once the number of messages in the incoming queue meets or drops below another threshold, the server can allocate the resources to the accepted queue. Allocate resources in this order to prevent significant delays and timeouts for unaccepted messages. Once the number of messages in the accepted queue meets or drops below another specified threshold, incoming messages can be added to the incoming queue again as processing returns to normal.

[0009] The incoming queue may also meet or exceed the threshold number of messages. In some embodiments, the server responds by rejecting subsequent messages from the incoming queue. Additionally, the server can determine whether the number of messages in the accepted queue is low (either in absolute terms or relative to the number of messages in the incoming queue). If the number of messages in the accepted queue is low, other resources can be allocated to the incoming queue.

[0010] Although embodiments with two queues are discussed herein, embodiments with other queues can be similarly envisioned. Additionally, although embodiments where messages are allocated to queues based on acceptance are discussed herein, other classifications (e.g., by request type, request priority, etc.) can be similarly considered. Furthermore, although receiving and processing messages are referenced, other targets (e.g., requests, tasks, etc.) can be similarly envisioned.

[0011] It should be understood that various aspects of the subject matter briefly described above and described in further detail below can be implemented as a hardware device, a computer-implemented method, a computer-controlled apparatus or device, a computing system, or an article of manufacture (e.g., a computer storage medium). Although the subject matter described herein is presented in the general context of program modules executed on one or more computing devices, those of ordinary skill in the art should recognize that other implementations can be performed in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types.

[0012] Those of ordinary skill in the art should also understand that aspects of the subject matter described herein can be practiced on or in conjunction with other computer system configurations other than those specifically described herein, which other computer systems include multiprocessor systems, microprocessor-based or programmable consumer electronics, AR, VR, and MR devices, video game devices, handheld computers, smartphones, smart TVs, autonomous vehicles, smartwatches, e-readers, tablet computing devices, dedicated hardware devices, network devices, and the like.

[0013] By reading the following detailed description and viewing the associated drawings, other functions and advantages beyond those explicitly described above will become apparent. This general section is provided to introduce in a simplified form a selection of concepts that will be further described in the following detailed description. This general section is not intended to identify key features or essential features of the invention, nor is it used to limit the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a computing system diagram showing aspects of an operating environment for embodiments disclosed herein, the operating environment including an asymmetric cooperative queue for processing incoming messages.

[0015] Figure 2 is a computing system diagram showing an asymmetric, cooperative queue when the number of messages in the accepted queue meets or exceeds a threshold.

[0016] Figure 3 is a computing system diagram showing a response to a threshold number of messages in the accepted queue being met or exceeded.

[0017] Figure 4 is a computing system diagram showing how messages are preferentially processed from an incoming queue in response to exceeding a threshold number of messages in the receive queue.

[0018] Figure 5 is a computing system diagram showing preferentially processing messages from the accepted queue after the incoming queue has been exhausted.

[0019] Figure 6 is a computational system diagram showing the processing of messages from an accepted queue until another threshold is exceeded.

[0020] Figure 7 is a computational system diagram showing the number of messages in an accepted queue dropping below another threshold.

[0021] Figure 8 is a computational system diagram showing the response to the number of messages in a receiving queue dropping below another threshold.

[0022] Figure 9 is a computational system diagram showing the number of messages in an incoming queue being higher than a specified threshold.

[0023] Figure 10 is a computational system diagram showing the response to the number of messages in an incoming queue rising above the specified threshold.

[0024] Figure 11 is a computational system diagram showing the processing of messages after responding to the number of messages in an incoming queue rising above the specified threshold.

[0025] Figure 12 is a computational system diagram showing the processing returning to a normal state after the number of messages in an incoming queue drops below another threshold.

[0026] Figure 13 shows aspects of routines for implementing aspects of the techniques disclosed herein as shown and described below.

[0027] Figure 14 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computational system capable of implementing the techniques and aspects of the techniques presented herein. Detailed Description

[0028] The following detailed description describes improved techniques for a server to process a large number of messages. As described above, before the server processes a message, an asymmetric cooperative queue is used to store the message (also referred to as a request, task, etc.). These queues are asymmetric because they store different types of messages. For example, one queue can store incoming messages received from a message generator (before the server accepts the message), while another queue can store messages that have been accepted for processing. In this way, messages can be processed in stages, for example, passing through two queues before being fully processed.

[0029] These queues can also be considered asymmetric because the message generators have different expectations based on the queue in which the message resides. Messages received from the message generators and stored in the incoming queue may be discarded (i.e., not processed), so there is no expectation that the message will be processed. By not expecting the message to be processed, the message generators can monitor whether the message fails or progresses to the accepted queue and retry the message if it fails. At the same time, once the message is added to the accepted queue for processing, the message generators will expect the message to be processed, enabling the server to handle failures, retries, and so on.

[0030] In some embodiments, the asymmetric queues are cooperative because they use a common pool of processors, memory, network bandwidth, and other computing resources and because they cede computing resources to each other when needed. In some embodiments, computing resources are allocated proportionally to the number of items in each queue or with a bias towards the accepted queue (e.g., an offset of 1 - 10% or more). However, as described below, the computing resource allocation may change in response to a large number of messages.

[0031] In some embodiments, priorities are assigned to each queue and, in part based on these priorities, a large number of messages are responded to. For example, the incoming queue may have a lower priority (also referred to as 'non - priority'), while the accepted queue may have a higher priority (also referred to as 'priority'). Details of how queues with different priorities are processed in response to a large number of messages are described below.

[0032] In some embodiments, each message is associated with a customer (e.g., a client, a customer account, a tenant, etc.). Each customer can be associated with a quota of pending I / O (e.g., disk bandwidth) that will use the messages in the incoming and received queues during processing (quota of input / output). The quota can be based on the number of individuals in the customer organization, the amount the customer has paid, the priority associated with the customer, the number of customers assigned to a particular server, and so on. The quota can be an absolute number, such as a number of megabytes, a certain rate (e.g., number of kilobytes per second, percentage of available I / O bandwidth on a given server, etc.).

[0033] In some embodiments, when a message is received from the message generator, it is determined whether the customer associated with the message has exceeded its quota of pending I / O. For example, the message processing engine can determine whether the received message causes the pending I / O of the associated customer to exceed its quota, and if so, the message will be rejected from entering the incoming queue.

[0034] Additionally or alternatively, it can be determined whether the computing resources of the server are overloaded. This can occur when the system is set to achieve maximum throughput, thereby subjecting the CPU and other resources to near-constant stress. For example, it can be determined that the number or percentage of CPU cycles dedicated to message processing has exceeded a specified threshold. Other determinations include: detecting an increase in CPU temperature, high power consumption, and so on. Once it is determined that the computing resources of the server are overloaded, in one embodiment, the processing of messages is paused while discarding new messages for the incoming queue.

[0035] Figure 1 is a computing system diagram showing aspects of an operating environment for embodiments disclosed herein, the operating environment including an asymmetric cooperative queue for processing incoming messages. As Figure 1 shown, system 100 discloses a message generator 102 executing on a client computer 101, which communicates with a message processing engine 104 executing on a server computer 103 via a network 106. Although Figure 1 a single message generator 102 and message processing engine 104 are depicted, multiple message generators and message processing engines executing on the same and / or multiple computing devices can be similarly expected.

[0036] In some embodiments, the incoming queue 108 includes a first-in, first-out (FIFO) data structure that stores messages 109 received from the message generator 102. The incoming queue 108 stores the messages 114 until the CPU is available to process them. The message generator 102 has no expectation regarding the messages stored in the incoming queue (the message can be successfully processed to completion, discarded, timed out, etc.). If the message fails anyway, the message generator 102 can resubmit the message, submit the message to a different server, or otherwise handle the failure.

[0037] The incoming queue 108 can provide queued messages (e.g., message 111) to one or more of the processors (CPUs) 112A - 112D. In some embodiments, the processor will immediately process the message. However, in other embodiments, processing an incoming message can include: accepting the message (or a new message derived therefrom, such as message 115) and copying it into the accepted queue 110 for further processing.

[0038] In some embodiments, the accepted queue 110 stores accepted messages 116. The same pool of processing resources (e.g., CPUs 112A - 112D) can be used to process the messages 116. Processing resources can be allocated between the incoming queue 108 and the accepted queue 110 based on a variety of factors described throughout this document. However, briefly, when the message processing engine 104 is in a normal state, i.e., when neither the incoming queue 108 nor the accepted queue 110 has more than a specified threshold number of items, one embodiment allocates processing resources in proportion to the number of messages in each queue. In Figure 1 this, the allocation is depicted by slider 122.

[0039] In some embodiments, each queue has a threshold number of messages that can be stored before triggering a response. The thresholds for each queue can be the same or different. These thresholds can be based in part on the amount of computing resources available for processing messages. For example, the threshold for a server with 4 CPUs may be lower than the threshold for a server with 8 CPUs. The threshold can also be based on the time it takes to process a message, and that amount of time depends on computing power and task complexity. Based on feedback regarding server utilization, the threshold can be dynamic. For example, the threshold can be increased when the server 103 is frequently idle, or decreased when it is determined that the server 103 has been running at full capacity for too long. The threshold can also be user - defined, set to a default value, or extracted from a configuration file. As shown, the threshold 118 is associated with the incoming queue 108, and the threshold 120 is associated with the accepted queue 110, and neither threshold is met.

[0040] Figure 2 FIG. 200 is a computing system diagram showing an asymmetric cooperative queue when the number of messages in the accepted queue exceeds a threshold. Specifically, Figure 2 it depicts that a message 202 has been accepted and placed in the accepted queue 110 as message 204.

[0041] Figure 3 FIG. 300 is a computing system diagram showing a response to exceeding the threshold number of messages in the accepted queue. In some embodiments, the response includes: rejecting incoming messages from entering the incoming queue 108, as depicted by block 302. Additionally or alternatively, computing resources can be allocated substantially or fully to processing messages 114 in queue 108, as indicated by the movement of slider 122 towards the incoming queue 108.

[0042] Figure 4FIG. 400 is a computing system diagram showing how messages are preferentially processed from an incoming queue in response to exceeding a threshold number of messages in the receive queue. For example, message handler 104 has received message 114, as indicated by message profile 402 in incoming queue 108 and other messages 404 in the received queue 110.

[0043] Figure 5 FIG. 500 is a computing system diagram showing preferentially processing messages from a received queue after the incoming queue has been exhausted. In some embodiments, once incoming queue 108 has been exhausted, computing resources may be substantially or fully allocated to processing messages from received queue 110, as depicted by slider 122 moving towards received queue 110.

[0044] Figure 6 FIG. 600 is a computing system diagram showing processing messages from a received queue until another threshold is exceeded. In some embodiments, many messages 116 have been processed and the number of messages in received queue 110 has dropped below threshold 120. In some embodiments, processing may return to a normal state from this state, i.e., the trigger to return to the normal processing state drops below threshold 120. However, in other embodiments, processing of messages in received queue 110 may continue until received queue 110 is exhausted, at which point message handler 104 may return to the normal processing state.

[0045] Figure 7 FIG. 700 is a computing system diagram showing the number of messages in a received queue dropping below another threshold. In some embodiments, after the number of messages in received queue 110 drops below threshold 602, message handler 104 returns to the normal processing state.

[0046] Figure 8 FIG. 800 is a computing system diagram showing a response to the number of messages in a receive queue dropping below another threshold 602. In some embodiments, box 302 is removed, allowing new messages to enter incoming queue 108. At the same time, slider 122 reflects a rebalancing of computing resources away from substantial or full allocation to received queue 110 due to the expectation that new messages will arrive and be processed from incoming queue 108.

[0047] Figure 9Computing system diagram 900 shows that the number of messages in the incoming queue is higher than a specified threshold. In some embodiments, slider 122 reflects that 70% of the computing resources are allocated to incoming queue 108 because incoming queue 108 stores approximately 70% of the messages (10 out of 14) in any queue. One of messages 404 has not been processed, and within the time it took for message 1002 to be pushed onto incoming queue 108, three messages 1004 have been received.

[0048] Figure 10 Computing system diagram 1000 shows a response to the number of messages in the incoming queue rising above the specified threshold. In some embodiments, in response to incoming queue 108 being overwhelmed, block 1002 rejects additional incoming messages. By blocking incoming messages, server 104 is given time to process the existing queued messages. In some embodiments, if the number of messages in accepted queue 110 is relatively low compared to the number of messages in incoming queue 108 (e.g., incoming queue 108 has 2x, 3x, 4x, 10x, or 20x more messages than accepted queue 110, but other ratios can be similarly expected), then the processing resources can be biased towards incoming queue 108.

[0049] Figure 11 Computing system diagram 1100 shows the processing of messages after responding to the number of messages in the incoming queue rising above the specified threshold. In some embodiments, the number of messages in incoming queue 108 has decreased below a second threshold 122, triggering the incoming queue to reopen. At the same time, certain messages 1002 have been received into accepted queue 110 as messages 1102.

[0050] Figure 12 Computing system diagram 1200 shows that after the number of messages in the incoming queue drops below a second threshold 122, the processing returns to the normal state. In some embodiments, block 1002 has been removed and additional messages can be received and processed.

[0051] Figure 13 Aspects of routine 1300 are shown, which is used to implement various aspects of the techniques disclosed herein as shown and described above. Those of ordinary skill in the art should understand that the operations of the methods disclosed herein are not presented in any particular order, and it is possible and expected to perform some or all of these operations in an alternative order. For ease of description and illustration, these operations have been presented in a demonstration order. Operations can be added, omitted, performed together, and / or simultaneously without departing from the scope of the appended claims.

[0052] It should also be understood that the methods shown may end at any time and need not be executed in their entirety. Some or all of the operations of the method, and / or substantially equivalent operations, may be performed by executing computer-readable instructions included on a computer storage medium, as set forth herein. The term "computer-readable instructions" and variations thereof, as used in the specification and claims, are used herein broadly to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. The computer-readable instructions may be implemented on a variety of system configurations including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, hand-held computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0053] Accordingly, it should be understood that the logical operations described herein are implemented as: (1) sequences of computer-implemented actions or program modules running on a computing system (e.g., those described herein); and / or (2) interconnected machine logic circuits or circuit modules within a computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Accordingly, these logical operations may be implemented in software, firmware, special purpose digital logic, and any combination thereof.

[0054] In addition, the operations described above with reference to Figures 1 to 12 the exemplary computing devices described Figure 13 and shown in other figures may be implemented. For example, the various devices and / or modules described herein may receive, accept, and / or process messages and manage message processing based on computing resource usage, queue capacity, and / or thresholds.

[0055] Routine 1300 begins at operation 1302, where at operation 1302, a message processing engine 104 executing on a server computing device 103 receives a plurality of messages into an incoming queue 108. In some embodiments, the messages include electronic mail and the message processing engine 104 includes a message server (e.g., an electronic mail server). However, other message types may be similarly contemplated, such as Web requests, Web service requests, text messages, instant messaging messages, database requests, file system requests, or any other type of message sent to a computer. In some embodiments, the messages are received from a message generator such as a smart phone, a tablet device, a desktop computer, or any other network-enabled computing device. Routine 1300 then proceeds from operation 1302 to operation 1304.

[0056] At operation 1304, the message processing engine 104 processes one or more messages from the incoming queue. In some embodiments, each message is processed by one of a set of processing threads, processors, services, or types of processing devices (e.g., CPU 112A - 112D). In some embodiments, processing a message from the incoming queue includes: accepting the message. Accepting the message can include: explicitly or implicitly indicating to the calling process that the message has been received and will be processed. In some embodiments, accepting the message can include: adding the accepted message to the accepted queue 110. In some embodiments, the calling process (e.g., message generator 102) may expect the accepted message to be processed. In some embodiments, the accepted queue is given a high priority, or otherwise preferred.

[0057] Then, routine 1300 proceeds to operation 1306, where at operation 1306, the message processing engine 104 processes one or more messages from the accepted queue 110. In some embodiments, the same set of processing devices used to process messages from the incoming queue 108 are used to process messages from the accepted queue 110.

[0058] Then, routine 1300 proceeds to operation 1308, where the processing message engine 104 allocates resources between the incoming queue 108 and the accepted queue 110. In some embodiments, computing resources are allocated based on the relative number of messages in each queue. However, in some embodiments, to avoid overwhelming the accepted queue 110, processing resources may be biased towards the accepted queue 110.

[0059] Then, routine 1300 proceeds to operation 1310, where the message processing engine 104 determines whether the number of messages in the accepted queue meets or exceeds a specified threshold. Determining whether the number of messages in the accepted queue meets or exceeds a specified threshold can determine whether the server computing device 103 is approaching its capacity. Additionally or alternatively, determining that the number of messages in the accepted queue meets or exceeds a specified threshold can indicate that the message processing latency is too high.

[0060] In some embodiments, the threshold may include an absolute number or a percentage of the maximum queue size. In some embodiments, the threshold may be determined dynamically based on the measured throughput of the message processing engine. For example, the message processing engine may attempt different thresholds and measure the resulting throughput level (e.g., the number of messages processed per second), and then select the threshold that results in the highest throughput. In some embodiments, the threshold may be selected dynamically to minimize message processing latency (e.g., by attempting different thresholds, measuring the average message throughput, and selecting the threshold that results in the lowest latency). In some embodiments, the threshold may be selected based on a combination of these and other factors, balancing user-defined preferences for low latency, high throughput, or other metrics (e.g., energy efficiency, propensity to crash, cooling metrics, or other aspects of operating the server computing device 103).

[0061] Then, routine 1300 proceeds to operation 1312, where the message processing engine 104 blocks subsequent messages from being added to the incoming queue 108 in response to determining that the number of messages in the receive queue exceeds a specified threshold. By blocking messages from being added to the incoming queue 108 but allowing messages to continue to be added to the accepted queue 110, the message processing engine 104 creates breathing room that allows existing messages to be processed without interruption.

[0062] In some embodiments, messages are blocked from being added to the incoming queue 108 by discarding them (i.e., not responding to them). In other embodiments, the message processing engine 104 may return an error code, e.g., an indication that the server is full and temporarily unable to accept new messages for processing.

[0063] Then, routine 1300 proceeds to operation 1314, where, at operation 1314, the message processing engine 104 diverts computing resources to process the incoming queue 108. In some embodiments, all or substantially all of the computing resources are allocated to process messages from the incoming queue 108, leaving little or no resources to process messages from the accepted queue 110. In this way, messages are in the incoming queue. However, in other embodiments, resources may be allocated between the incoming queue 108 and the accepted queue 110 in proportion to the number of messages stored in each queue, whether or not biased towards the incoming queue 108.

[0064] Then, routine 1300 proceeds to operation 1316, where, at operation 1316, the message processing engine 104 diverts computing resources to process the accepted queue 110. In some embodiments, all or substantially all of the computing resources are diverted to process the accepted queue 110 in order to process the messages in the accepted queue 110 as quickly as possible.

[0065] Then, routine 1300 proceeds to operation 1318, where, at operation 1318, the message processing engine 104 determines whether the number of messages stored in the accepted queue 110 has dropped below a second threshold. In some embodiments, the second threshold is defined as an absolute number of messages, a percentage of the first threshold (e.g., 50% of the first threshold), and so on. Additionally or alternatively, the second threshold may be defined dynamically based on feedback from the message processing engine. For example, when different second thresholds are employed, overall throughput, message latency, and other metrics may be evaluated, and one or a combination of these factors may be used to select the second prescribed threshold. Routine 1300 proceeds from operation 1318 to operation 1320, where it ends.

[0066] It should be understood that the subject matter described above may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as a computer-readable storage medium. Operations of example methods are illustrated in various boxes and are summarized with reference to these boxes. The methods are shown as a logical flow of boxes, each box representing one or more operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the recited operations.

[0067] In general, computer-executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order of description of the operations should not be construed as limiting, and any number of the described operations may be executed in any order, combined in any order, subdivided into multiple sub-operations, and / or executed in parallel to implement the described processing. The described processing may be performed by resources associated with one or more devices (e.g., one or more internal or external CPUs or GPUs) and / or one or more hardware logics (e.g., field programmable gate arrays (“FPGAs”), digital signal processors (“DSPs”), or other types of accelerators).

[0068] All of the methods and processes described above may be embodied and fully automated by software code modules executed by one or more general-purpose computers or processors. These code modules may be stored in any type of computer-readable storage medium or other computer storage device (e.g., those described below). Some or all of these methods may alternatively be embodied in dedicated computer hardware (e.g., those described below).

[0069] Any routine description, element, or box in the flowcharts described herein and / or depicted in the accompanying figures should be understood as potentially representing a part of a module, segment, or code that includes one or more executable instructions for implementing specific logical functions or elements in a routine. Alternative implementations are within the scope of the examples described herein, where elements or functions may be deleted or may be executed in a different order than shown or discussed, including substantially synchronously or in the reverse order, as would be understood by one of ordinary skill in the art.

[0070] Figure 14 Other details of an exemplary computer architecture 1400 for a computer (e.g., the server computer 103 that executes the message handling engine 104) capable of executing the program components described herein are shown. Thus, Figure 14 The computer architecture 1400 shown in depicts the architecture for a server computer, mobile phone, PDA, smart phone, desktop computer, netbook computer, tablet computer, and / or laptop computer. The computer architecture 1400 can be utilized to execute any aspect of the software components presented herein.

[0071] Figure 14 The computer architecture 1400 shown includes a central processing unit 1402 (“CPU”), a system memory 1404 (wherein the system memory 1404 includes a random access memory 1406 (“RAM”) and a read-only memory (“ROM”) 1408), and a system bus 1410 that couples the memory 1404 to the CPU 1402. A basic input / output system is stored in the ROM 1408, which contains basic routines that assist in passing information between the various elements in the computer architecture 1400 (e.g., during startup). The computer architecture 1400 also includes a mass storage device 1412 for storing an operating system 1407, other data, and one or more message handling engines 104.

[0072] The mass storage device 1412 is connected to the CPU 1402 through a mass storage controller (not shown) connected to the bus 1410. The mass storage device 1412 and its associated computer-readable medium provide non-volatile storage for the computer architecture 1400. Although the description of the computer-readable medium included herein refers to a mass storage device (e.g., a solid state drive, hard disk, or CD-ROM drive), one of ordinary skill in the art should understand that the computer-readable medium can be any available computer storage medium or communication medium accessible by the computer architecture 1400.

[0073] A communication medium includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any transmission medium. The term "modulated data signal" means a signal that has one or more of its characteristics changed or set in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic waves, RF, infrared, and other wireless media. Any combination of the foregoing should also be included within the scope of computer-readable media.

[0074] By way of example, and not limitation, computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, computer media includes but is not limited to RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technologies, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computer architecture 1400. For purposes of facilitating the claims, the phrase "computer storage medium", "computer-readable storage medium", and variations thereof do not themselves include wave forms, signals, and / or other transitory and / or intangible communication media.

[0075] In accordance with various configurations, the computer architecture 1400 can operate in a networked environment using logical connections to remote computers through the network 1456 and / or another network (not shown). The computer architecture 1400 can be connected to the network 1456 through a network interface unit 1414 connected to the bus 1410. It should be understood that the network interface unit 1414 can also be used to connect to other types of networks and remote computer systems. The computer architecture 1400 can also include an input / output controller 1416 for receiving and processing input from a plurality of other devices including a keyboard, mouse, or electronic stylus ( Figure 14 not shown in the figure). Similarly, the input / output controller 1416 can provide output to a display screen, printer, or other type of output device ( Figure 14 also not shown in the figure).

[0076] It should be understood that the software components described herein, when loaded into and executed by the CPU 1402, can transform the CPU 1402 and the entire computer architecture 1400 from a general-purpose computing system into a special-purpose computing system customized to facilitate the functions presented herein. The CPU 1402 can be composed of any number of transistors or other discrete circuit elements, which can individually or jointly exhibit any number of states. Specifically, the CPU 1402 can operate as a finite state machine in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions can transform the CPU 1402 by specifying how the CPU 1402 transitions between states, thereby transforming the transistors or other discrete hardware elements that make up the CPU 1402.

[0077] Encoding the software modules given herein can also transform the physical structure of the computer-readable medium given herein. In different embodiments of this specification, the specific transformation of the physical structure can depend on various factors. Examples of such factors can include, but are not limited to: the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as main memory or auxiliary memory, and so on. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of the transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. The software can also transform the physical state of these components in order to store data thereon.

[0078] As another example, the computer-readable medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when the software given herein is encoded in the magnetic or optical medium, the software can transform the physical state of the magnetic or optical medium. These transformations can include changing the magnetic properties of a specific location within a given magnetic medium. These transformations can also include changing the physical characteristics or properties of a specific location within a given optical medium in order to change the optical properties of those locations. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification, and the foregoing examples are merely helpful in facilitating the discussion.

[0079] Based on the foregoing, it should be understood that many types of physical transformations occur in the computer architecture 1400 in order to store and execute the software components presented herein. It should also be understood that the computer architecture 1400 can include other types of computing devices, which include handheld computers, embedded computer systems, personal digital assistants, and other types of computing devices known to those of ordinary skill in the art. It is also contemplated that the computer architecture 1400 may not include Figure 14 all of the components shown in Figure 14other components explicitly shown in, or may utilize an architecture that is completely different from the architecture shown in Figure 14 the architecture shown in

[0080] Exemplary Clauses

[0081] The disclosure presented herein covers the subject matter set forth in the following exemplary clauses.

[0082] Example 1: A system includes: one or more data processing units (1402); a computer-readable medium (1404) encoded with computer-executable instructions (104) to cause the one or more data processing units (1402) to: receive a plurality of messages (113) into an incoming queue (108); add at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); determine that the number of messages in the accepted queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the accepted queue (110) exceeds the threshold number (120): block (302) subsequent messages from entering the incoming queue (108); transfer computing resources (112) to process messages in the incoming queue (108); in response to a message from the incoming queue (108) being processed, allocate computing resources (112) to process messages from the accepted queue (110); remove the block (302) to allow messages to enter the incoming queue (108).

[0083] Example 2: The system according to Example 1, wherein messages from the accepted queue (110) are processed until the accepted queue (110) is emptied.

[0084] Example 3: The system according to Example 1, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0085] Example 4: The system according to Example 1, wherein when blocking subsequent messages from the incoming queue (108), messages are allowed to be added to the accepted queue (110).

[0086] Example 5: The system according to Example 1, wherein adding at least one message (115) of the plurality of messages to the accepted queue (110) includes: using the shared computing resources (112) to process messages (114) stored in the incoming queue (108).

[0087] Example 6: The system according to Example 1, wherein the computer-executable instructions (104) further cause the one or more data processing units (1402) to: determine when the number of messages in the incoming queue (108) exceeds an incoming queue threshold (118); in response to determining that the number of messages in the incoming queue (108) exceeds the incoming queue threshold (118), block (302) subsequent messages from entering the incoming queue (108); determine whether the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue; and when the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110), bias computing resources towards the incoming queue (108).

[0088] Example 7: The system according to Example 1, wherein each message is associated with a customer account, wherein the customer account has a quota of outstanding input / output (IO) operations, and wherein messages from customers who have exceeded the quota of outstanding IO are prohibited from adding messages to the incoming queue (108).

[0089] Example 8: The system according to Example 1, wherein the quota of outstanding IO operations is calculated based on the estimated amount of IO for completing the messages in the incoming queue (108) and the accepted queue (110).

[0090] Example 9: The system according to Example 1, wherein when the incoming queue (108) and the accepted queue (110) do not exceed their respective thresholds, shared computing resources are allocated in proportion to the number of messages in each queue.

[0091] Example 10: The system according to Example 9, wherein the shared computing resources are biased towards the accepted queue (110).

[0092] Example 11: A system includes: one or more data processing units (1402); a computer-readable medium (1404) having computer-executable instructions (104) encoded thereon to cause the one or more data processing units (1402) to: receive a plurality of messages (113) into an incoming queue (108); add at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); determine that the number of messages in the accepted queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the accepted queue (110) exceeds the threshold number (120): block (302) subsequent messages from entering the incoming queue (108); in response to the messages from the incoming queue (108) being processed, allocate computing resources (112) to process messages from the accepted queue (110); remove the block (302) to allow messages to enter the incoming queue (108).

[0093] Example 12: The system according to Example 11, wherein messages from the accepted queue (110) are processed until the accepted queue (110) is emptied.

[0094] Example 13: The system according to Example 11, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0095] Example 14: The system according to Example 11, wherein when blocking subsequent messages from the incoming queue (108), adding messages to the accepted queue (110) is allowed.

[0096] Example 15: A method employed by a computing device includes: receiving a plurality of messages (113) into an incoming queue (108); adding at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); determining that the number of messages in the accepted queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the accepted queue (110) exceeds the threshold number (120), for: blocking (302) subsequent messages from entering the incoming queue (108); in response to the messages from the incoming queue (108) being processed, allocating computing resources (112) to process messages from the accepted queue (110); removing the block (302) to allow messages to enter the incoming queue (108).

[0097] Example 16: The method according to Example 15 further includes: detecting when the computing resource 112 is overloaded, and in response to detecting that the computing resource 112 is overloaded, stopping processing messages from the incoming queue (108) and the accepted queue (110).

[0098] Example 17: The method according to Example 15 further includes: determining when the number of messages in the incoming queue (108) exceeds an incoming queue threshold (118); in response to determining that the number of messages in the incoming queue (108) exceeds the incoming queue threshold (118), for: blocking (302) subsequent messages from entering the incoming queue (108); determining whether the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110); and when the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110), biasing the computing resource towards the incoming queue (108).

[0099] Example 18: The method according to Example 15, wherein each message is associated with a customer account, wherein the customer account has a quota of outstanding input / output (IO) operations, and wherein messages from customers who have exceeded the quota of outstanding IO are prohibited from adding messages to the incoming queue (108).

[0100] Example 19: The method according to Example 15, wherein the quota of outstanding IO operations is calculated based on the estimated amount of IO for completing the messages in the incoming queue (108) and the accepted queue (110).

[0101] Example 20: The method according to Example 15, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0102] Compared with many other technical advantages, the technology herein can utilize computing resources such as processor cycles, memory, network bandwidth, and power consumption more efficiently compared to previous solutions that flood server computers with messages. Another technical advantage is to reduce message processing latency and avoid message timeouts when the server is overburdened. Other technical benefits not specifically mentioned herein can also be achieved through embodiments of the disclosed subject matter.

[0103] Although these technologies are described using language specific to structural features and / or method acts, it should be understood that the appended claims are not necessarily limited to these specific features or acts described above. Instead, these features and acts are described as exemplary embodiments of these technologies.

Claims

1. A system, comprising: one or more data processing units; and a computer-readable medium having computer-executable instructions encoded thereon to cause the system to: receive a plurality of tasks into an incoming queue; add at least one of the plurality of tasks to an accepted queue, wherein the incoming queue and the accepted queue share computing resources; determine that the number of tasks in the accepted queue exceeds a threshold number; in response to determining that the number of tasks in the accepted queue exceeds the threshold number: block subsequent tasks from entering the incoming queue; in response to a task from the incoming queue being processed, allocate computing resources for processing tasks from the accepted queue; and remove the block to allow tasks to enter the incoming queue.

2. The system according to claim 1, wherein, Process tasks from the accepted queue until the accepted queue has been exhausted.

3. The system according to claim 1, wherein Process tasks from the accepted queue until the number of tasks in the accepted queue drops below a second threshold.

4. The system according to claim 1, wherein, While blocking subsequent messages from the incoming queue, allow tasks to be added to the accepted queue.

5. The system according to claim 1, wherein Adding at least one of the plurality of tasks to the accepted queue includes: processing tasks stored in the incoming queue using shared computing resources.

6. The system according to claim 1, wherein The computer-executable instructions further cause the one or more data processing units to: determine when the number of tasks in the incoming queue exceeds an incoming queue threshold; in response to determining that the number of tasks in the incoming queue exceeds the incoming queue threshold: block subsequent tasks from entering the incoming queue; determine whether the number of tasks in the incoming queue significantly exceeds the number of tasks in the accepted queue; and when the number of tasks in the incoming queue significantly exceeds the number of tasks in the accepted queue, bias computing resources towards the incoming queue.

7. The system according to claim 1, wherein, Each task is associated with a customer account, wherein the customer account has a quota of outstanding input / output (IO) operations, and wherein tasks from customers that have exceeded the quota of outstanding IO operations are prohibited from adding tasks to the incoming queue.

8. The system according to claim 7, wherein, The quota of outstanding IO operations is calculated based on the estimated amount of IO for completing tasks in the incoming queue and the accepted queue.

9. The system according to claim 1, wherein When the incoming queue and the accepted queue do not exceed their respective thresholds, allocate shared computing resources proportionally to the number of tasks in each queue.

10. The system according to claim 9, wherein, Shared computing resources are biased towards the accepted queue.

11. A computer storage device having computer-executable instructions encoded thereon, the computer-executable instructions when executed by a processor of a computing device cause the computing device to: receive a plurality of tasks into an incoming queue; add at least one of the plurality of tasks to an accepted queue, wherein the incoming queue and the accepted queue share computing resources; determine that the number of tasks in the accepted queue exceeds a threshold number; in response to determining that the number of tasks in the accepted queue exceeds the threshold number: block subsequent tasks from entering the incoming queue; Upon processing of the task from the incoming queue, allocate computing resources for processing tasks from the accepted queue; and Remove the block to allow tasks to enter the incoming queue.

12. The computer storage device according to claim 11, wherein, Process tasks from the accepted queue until the accepted queue has been exhausted.

13. The computer storage device according to claim 11, wherein, Process tasks from the accepted queue until the number of tasks in the accepted queue drops below a second threshold.

14. The computer storage device according to claim 11, wherein, While blocking subsequent tasks from the incoming queue, allow tasks to be added to the accepted queue.

15. A method performed by a computing device, comprising: Receive a plurality of tasks into an incoming queue; Add at least one task of the plurality of tasks to an accepted queue, wherein the incoming queue and the accepted queue share computing resources; Determine that the number of tasks in the accepted queue exceeds a threshold number; In response to determining that the number of tasks in the accepted queue exceeds the threshold number: Block subsequent tasks from entering the incoming queue; Upon processing of the task from the incoming queue, allocate computing resources for processing tasks from the accepted queue; and Remove the block to allow tasks to enter the incoming queue.

16. The method according to claim 15 further comprises: Detect when the computing resources are overloaded, and in response to detecting that the computing resources are overloaded, stop processing tasks from both the incoming queue and the accepted queue.

17. The method according to claim 15, further comprising: Determine when the number of tasks in the incoming queue exceeds an incoming queue threshold; In response to determining that the number of tasks in the incoming queue exceeds the incoming queue threshold: Block subsequent tasks from entering the incoming queue; Determine whether the number of tasks in the incoming queue significantly exceeds the number of tasks in the accepted queue; and When the number of tasks in the incoming queue significantly exceeds the number of tasks in the accepted queue, bias the computing resources towards the incoming queue.

18. The method according to claim 15, wherein, Each task is associated with a customer account, wherein the customer account has a quota of outstanding input / output (IO) operations, and wherein tasks from customers that have exceeded the quota of outstanding IO operations are prohibited from adding tasks to the incoming queue.

19. The method according to claim 15, wherein, The quota of outstanding IO operations is calculated based on the estimated amount of IO for completing tasks in the incoming queue and the accepted queue.

20. The method according to claim 15, wherein Process tasks from the accepted queue until the number of tasks in the accepted queue drops below a second threshold.

Images