Asymmetric cooperative queue management of messages

Abstract

Disclosed herein is an improved technique for responding to a server that is flooded with messages. Messages may be initially placed in an "incoming" queue, and once the server receives a corresponding message for processing, the message may be placed in an accepted queue, and an indication may be generated indicating that the message will be processed. When the number of messages in the accepted queue meets or exceeds a specified threshold, new messages addressed to the incoming queue may be rejected. In addition, computing resources may not be allocated to the incoming queue until the messages in the incoming queue are processed (or other conditions are met). Once the incoming queue is empty or other conditions are met, resources may be allocated to the accepted queue. Once the number of messages in the accepted queue meets or drops below a second threshold, new messages may be allowed to enter the incoming queue again.

Description

Background Art

[0001] Message servers handle a massive workload, collectively processing hundreds of billions of messages per day around the world. However, message servers can become overwhelmed when incoming messages exceed the server's ability to process those messages. Messages sent to an overwhelmed server often remain in a queue until they are processed, which causes delays. When the delay is too long, the message may time out, causing the sender of the message to retry the message. Furthermore, an overwhelmed message server may continue to receive new messages, increasing delays and timeouts. Furthermore, as more and more resources are dedicated to dispatching messages, the resulting backlog reduces computer efficiency, reducing the resources available to actually process the messages. As such, it is understandable that there is a continuing need for improved techniques for responding to servers that are overwhelmed with messages.

[0002] It is with respect to these and other considerations that the disclosure made herein is presented. Summary of the invention

[0003] Disclosed herein is an improved technique for responding to a server that is flooded with messages. Messages may initially be placed in an "incoming" queue. Once a server receives a corresponding message for processing, the message may be placed in an accepted queue and an indication may be generated indicating that the message will be processed. When the number of messages in the accepted queue meets or exceeds a specified threshold, new messages addressed to the incoming queue may be rejected. In addition, computing resources may not be allocated to the incoming queue until the messages in the incoming queue are processed (or other conditions are met). Once the incoming queue is empty or other conditions are met, resources may be allocated to the accepted queue. Once the number of messages in the accepted queue meets or drops below a second threshold, new messages may be allowed to enter the incoming queue again.

[0004] In some embodiments, messages such as emails are placed in an "incoming" queue. The incoming queue contains messages that have not been accepted by the server, and there is no guarantee that the incoming message will be processed. Once a message is accepted by the server, it can be removed from the incoming queue and placed in the "accepted" queue. Messages in the accepted queue are guaranteed to be processed by the server.

[0005] In some embodiments, the incoming queue and the accepted queue are cooperative - they share a common pool of computing resources (e.g., processors, memory, network bandwidth, etc.), and they yield these resources to each other based on various criteria. For example, under normal operating conditions (i.e., when neither queue exceeds a threshold number of messages), computing resources can be allocated in proportion to the size of each queue. However, in some embodiments, the accepted queue can be given more or slightly more than proportional resources to avoid accumulating accepted messages.

[0006] Additionally or alternatively, when the number of messages in the accepted queue meets or exceeds a threshold, the server may respond by preventing new messages from being added to the incoming queue. While the incoming queue is blocked, the server may continue to allow messages into the accepted queue. Continuing to allow messages into the accepted queue is counterintuitive because the response is triggered by the large number of messages in the accepted queue. Rejecting messages from the incoming queue, but allowing messages into the accepted queue, prevents a runaway situation where more new messages are received than can be processed. At the same time, messages already in the incoming queue may still be added to the accepted queue and processed.

[0007] At the same time, the server can allocate some or all of its computing resources to the incoming queue. Once the number of messages in the incoming queue meets or drops below another threshold, the server can allocate resources to the accepted queue. Resources are allocated in this order to prevent large delays and timeouts for unaccepted messages. Once the number of messages in the accepted queue meets or drops below another specified threshold, incoming messages can be added to the incoming queue again as processing resumes normal.

[0008] The incoming queue may also meet or exceed a threshold number of messages. In some embodiments, the server responds by rejecting subsequent messages from the incoming queue. In addition, the server may determine whether the number of messages in the queue received is low (either in absolute terms or relative to the number of messages in the incoming queue). If the number of messages in the queue received is small, other resources may be allocated to the incoming queue.

[0009] Although embodiments with two queues are discussed herein, embodiments with other queues are similarly contemplated. In addition, although embodiments are discussed herein where messages are assigned to queues based on acceptance, other classifications (e.g., by request type, request priority, etc.) are similarly contemplated. Furthermore, although reference is made to receiving and processing messages, other objectives (e.g., requests, tasks, etc.) are similarly contemplated.

[0010] It should be understood that various aspects of the subject matter briefly described above and described in further detail below may be implemented as hardware devices, computer-implemented methods, computer-controlled apparatus or devices, computing systems, or articles of manufacture (e.g., computer storage media). Although the subject matter described herein is presented in the general context of program modules executed on one or more computing devices, one of ordinary skill in the art will recognize that other implementations may be performed in conjunction with other types of program modules. Typically, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types.

[0011] It will also be appreciated by those of ordinary skill in the art that aspects of the subject matter described herein may be practiced on or in conjunction with other computer system configurations than those specifically described herein, including multi-processor systems, microprocessor-based or programmable consumer electronics, AR, VR and MR devices, video gaming devices, handheld computers, smartphones, smart TVs, self-driving cars, smart watches, e-readers, tablet computing devices, dedicated hardware devices, network devices, and the like.

[0012] By reading the following detailed description and viewing the associated drawings, other features and advantages beyond those explicitly described above will become apparent. This summary is provided to introduce in simplified form a selection of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the present invention, nor is it intended to limit the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a computing system diagram illustrating aspects of an operating environment for embodiments disclosed herein, including asymmetric cooperative queues for processing incoming messages.

[0014] Figure 2 is a computing system diagram illustrating asymmetric, cooperative queues when the number of messages in the queue accepted meets or exceeds a threshold.

[0015] Figure 3 is a diagram of a computing system illustrating a response to meeting or exceeding a threshold number of messages in an accepted queue.

[0016] Figure 4 is a diagram of a computing system illustrating how messages are prioritized from an incoming queue in response to exceeding a threshold number of messages in a receive queue.

[0017] Figure 5 is a diagram of a computing system that illustrates prioritizing the processing of messages from an accepted queue after an incoming queue has been exhausted.

[0018] Figure 6 is a diagram of a computing system that illustrates processing messages from an accepted queue until another threshold is exceeded.

[0019] Figure 7 is a diagram of a computing system illustrating a number of messages in an accepted queue falling below another threshold.

[0020] Figure 8is a diagram of a computing system illustrating a response to the number of messages in a receive queue falling below another threshold.

[0021] Fig. 9 is a diagram of a computing system illustrating that the number of messages in an incoming queue is above a specified threshold.

[0022] Fig.10 is a diagram of a computing system illustrating a response to the number of messages in an incoming queue rising above a specified threshold.

[0023] Fig.11 is a diagram of a computing system that illustrates processing messages in response to the number of messages in an incoming queue rising above a specified threshold.

[0024] Fig.12 is a diagram of a computing system showing processing returning to a normal state after the number of messages in an incoming queue drops below another threshold.

[0025] Fig.13 Aspects of routines for implementing various aspects of the techniques disclosed herein as shown in the figures and described below are shown.

[0026] Fig.14 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing system capable of implementing various aspects of the techniques and technologies presented herein. DETAILED DESCRIPTION

[0027] The following detailed description describes an improved technique for a server to process a large number of messages. As described above, asymmetric cooperative queues are used to store messages (also called requests, tasks, etc.) before the server processes the messages. These queues are asymmetric because they store different types of messages. For example, one queue can store incoming messages received from a message generator (before the server accepts the message), while another queue can store messages that have been accepted for processing. In this way, messages can be processed in stages, such as passing through two queues before being fully processed.

[0028] These queues can also be considered asymmetric because the message producer has different expectations based on which queue the message is placed in. A message received from a message producer and stored in an incoming queue may be discarded (i.e., not processed), so there is no expectation that the message will be processed. By not expecting that the message will be processed, the message producer can monitor whether the message fails or progresses to the accepted queue, and retry the message if it fails. At the same time, once the message is added to the accepted queue for processing, the message producer will expect the message to be processed, allowing the server to handle failures, retries, and so on.

[0029] In some embodiments, asymmetric queues are cooperative because they use a common pool of processors, memory, network bandwidth, and other computing resources, and because they yield computing resources to each other when needed. In some embodiments, computing resources are allocated proportionally to the number of items in each queue, or with a bias toward the accepted queue (e.g., 1-10% or more). However, as described below, the computing resource allocation may change in response to a large number of messages.

[0030] In some embodiments, a priority is assigned to each queue, and a large number of messages are responded to based in part on these priorities. For example, the priority of the incoming queue may be lower (also referred to as 'non-priority'), while the priority of the received queue may be higher (also referred to as 'priority'). The details of how queues of different priorities are handled in response to a large number of messages are described below.

[0031] In some embodiments, each message is associated with a customer (e.g., client, customer account, tenant, etc.). Each customer can be associated with a quota (input / output quota) of pending IO (e.g., disk bandwidth) that will be used when processing messages in the incoming and received queues. The quota can be based on the number of individuals in the customer organization, the amount of money the customer has paid, the priority associated with the customer, the number of customers assigned to a particular server, etc. The quota can be an absolute number, such as a number of megabytes, a rate (e.g., kilobytes per second, a percentage of the available IO bandwidth on a given server, etc.).

[0032] In some embodiments, when a message is received from a message generator, it is determined whether the client associated with the message exceeds its quota of pending IOs. For example, the message processing engine can determine whether the received message causes the pending IOs of the associated client to exceed its quota, and if so, the message will be rejected from entering the incoming queue.

[0033] Additionally or alternatively, it may be determined whether the computing resources of the server are overloaded. This may occur when the system is set to achieve maximum throughput, thereby placing the CPU and other resources under near-constant stress. For example, it may be determined that the number or percentage of CPU cycles dedicated to message processing has exceeded a specified threshold. Other determinations include: detecting an increase in CPU temperature, high power consumption, etc. Once it is determined that the computing resources of the server are overloaded, in one embodiment, the processing of messages is suspended while new messages for incoming queues are discarded.

[0034] Figure 1 is a computing system diagram illustrating aspects of an operating environment for embodiments disclosed herein, including an asymmetric cooperative queue for processing incoming messages. Figure 1As shown in FIG. 1 , system 100 discloses a message generator 102 executing on a client computer 101 that communicates with a message processing engine 104 executing on a server computer 103 via a network 106. Figure 1 A single message generator 102 and message processing engine 104 is depicted, but multiple message generators and message processing engines executing on the same and / or multiple computing devices are similarly contemplated.

[0035] In some embodiments, the incoming queue 108 includes a first-in, first-out (FIFO) data structure that stores messages 109 received from the message generator 102. The incoming queue 108 stores messages 114 until the CPU is available to process them. The message generator 102 has no expectations for the messages stored in the incoming queue 108 (the message may be successfully processed to completion, discarded, timed out, etc.). If the message fails anyway, the message generator 102 may resubmit the message, submit the message to a different server, or otherwise handle the failure.

[0036] The incoming queue 108 can provide queued messages (e.g., message 111) to one or more of the processors (CPUs) 112A-112D. In some embodiments, the processor will process the message immediately. However, in other embodiments, processing the incoming message may include accepting the message (or a new message derived therefrom, such as message 115) and copying it to the accepted queue 110 for further processing.

[0037] In some embodiments, the accepted queue 110 stores the accepted messages 116. The same pool of processing resources (e.g., CPUs 112A-112D) may be used to process the messages 116. The processing resources may be allocated between the incoming queue 108 and the accepted queue 110 based on a variety of factors described throughout this document. However, briefly, when the message processing engine 104 is in a normal state, that is, when neither the incoming queue 108 or the accepted queue 110 has more than a specified threshold number of items, one embodiment allocates processing resources in proportion to the number of messages in each queue. Figure 1 , the allocation is depicted by slider 122.

[0038] In some embodiments, each queue has a threshold number of messages that can be stored before a response is triggered. The thresholds for each queue may be the same or different. These thresholds may be based in part on the amount of computing resources available for processing messages. For example, the threshold for a server with 4 CPUs may be lower than the threshold for a server with 8 CPUs. The threshold may also be based on the time spent processing messages, which depends on computing power and task complexity. Based on feedback about server utilization, the threshold may be dynamic. For example, when server 103 is frequently idle, the threshold may be increased, or when determining that server 103 is running at full capacity for too long, the threshold may be lowered. The threshold may also be user-defined, which is set to a default value, or extracted from a configuration file. As shown, threshold 118 is associated with incoming queue 108, while threshold 120 is associated with accepted queue 110, and both thresholds are not met.

[0039] Figure 2 2 is a diagram of a computing system of an asymmetric cooperative queue when the number of messages in the queue received exceeds a threshold. Specifically, Figure 2 It is depicted that message 202 has been accepted and placed in accepted queue 110 as message 204 .

[0040] Figure 3 300 is a diagram of a computing system illustrating a response to exceeding a threshold number of messages in an accepted queue. In some embodiments, the response includes rejecting the incoming message from entering the incoming queue 108, as depicted by block 302. Additionally or alternatively, computing resources may be substantially or completely allocated to processing messages 114 in the queue 108, as indicated by the slider 122 moving toward the incoming queue 108.

[0041] Figure 4 400 is a diagram of a computing system showing how to prioritize messages from an incoming queue in response to exceeding a threshold number of messages in a received queue. For example, the message processing engine 104 has accepted the message 114, as indicated by the message profile 402 in the incoming queue 108 and other messages 404 located in the accepted queue 110.

[0042] Figure 5 5 is a diagram of a computing system 500 showing that after the incoming queue has been exhausted, messages from the accepted queue are prioritized for processing. In some embodiments, once the incoming queue 108 has been exhausted, computing resources can be substantially or completely allocated to processing messages from the accepted queue 110, as depicted by the slider 122 moving toward the accepted queue 110.

[0043] Figure 66 is a diagram of a computing system 600 showing processing of messages from an accepted queue until another threshold is exceeded. In some embodiments, a number of messages 116 have been processed and the number of messages in the accepted queue 110 drops below a threshold 120. In some embodiments, processing may return to a normal state from this state, i.e., the trigger for returning to the normal processing state falls below the threshold 120. However, in other embodiments, processing of messages in the accepted queue 110 may continue until the accepted queue 110 is exhausted, at which point the message processing engine 104 may return to the normal processing state.

[0044] Figure 7 700 is a computing system diagram showing the number of messages in the accepted queue falling below another threshold. In some embodiments, after the number of messages in the accepted queue 110 falls below the threshold 602, the message processing engine 104 returns to a normal processing state.

[0045] Figure 8 8 is a computing system diagram 800 showing a response to the number of messages in the received queue falling below another threshold 602. In some embodiments, block 302 is removed, allowing new messages to enter the incoming queue 108. At the same time, the slider 122 reflects the rebalancing of computing resources away from a substantial or full allocation to the accepted queue 110 in anticipation of new messages arriving and being processed from the incoming queue 108.

[0046] Fig. 9 9 is a diagram of a computing system showing that the number of messages in the incoming queue is above a specified threshold. In some embodiments, slider 122 reflects that 70% of the computing resources are allocated to incoming queue 108 because incoming queue 108 stores approximately 70% of the messages (10 out of 14) in any one queue. One of messages 404 has not yet been processed, and in the time it took for message 1002 to be pushed to incoming queue 108, three messages 1004 have been accepted.

[0047] Fig.10 1 is a computing system diagram 1000 illustrating a response to the number of messages in an incoming queue rising above a specified threshold. In some embodiments, in response to the incoming queue 108 being overwhelmed, block 1002 rejects additional incoming messages. By blocking incoming messages, the server 104 is given time to process existing queued messages. In some embodiments, if the number of messages in the accepted queue 110 is low relative to the number of messages in the incoming queue 108 (e.g., the incoming queue 108 has 2x, 3x, 4x, 10x, or 20x more messages than the accepted queue 110, but other ratios are similarly contemplated), then processing resources may be biased toward the incoming queue 108.

[0048] Fig.11 1100 is a diagram of a computing system that processes messages in response to the number of messages in the incoming queue rising above a specified threshold. In some embodiments, the number of messages in the incoming queue 108 has decreased below the second threshold 122, thereby triggering the incoming queue to be reopened. At the same time, some messages 1002 have been accepted into the accepted queue 110 as messages 1102.

[0049] Fig.12 is a computing system diagram 1200 showing processing returning to a normal state after the number of messages in the incoming queue drops below the second threshold 122. In some embodiments, block 1002 has been removed, and additional messages may be received and processed.

[0050] Fig.13 Various aspects of routine 1300 are shown, which is used to implement various aspects of the technology disclosed herein as shown and described above. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein are not presented in any particular order, and it is possible and expected to perform some or all of these operations in an alternative order. For ease of description and illustration, these operations have been introduced in the order of demonstration. Without departing from the scope of protection of the attached claims, operations can be added, omitted, performed together and / or simultaneously.

[0051] It should also be understood that the method shown can be terminated at any time and does not need to be performed in its entirety. Some or all of the operations of the method and / or substantially equivalent operations, as specified herein, can be performed by executing computer-readable instructions included on a computer storage medium. The term "computer-readable instructions" and its variants, as used in the specification and claims, are broadly used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions can be implemented on various system configurations including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0052] Therefore, it should be understood that the logical operations described herein are implemented as: (1) a computer-implemented sequence of actions or program modules running on a computing system (e.g., those described herein); and / or (2) interconnected machine logic circuits or circuit modules in a computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, these logical operations can be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0053] In addition, you can refer to the above Figures 1 to 12 The exemplary computing device described herein implements Fig.13 For example, various devices and / or modules described herein may receive, accept, and / or process messages, and manage message processing based on computing resource usage, queue capacity, and / or thresholds.

[0054] Routine 1300 begins at operation 1302, where at operation 1302, a message processing engine 104 executed on a server computing device 103 receives a plurality of messages into an incoming queue 108. In some embodiments, the message comprises an email, and the message processing engine 104 comprises a message server (e.g., an email server). However, other message types may also be similarly considered, such as web requests, web service requests, text messages, instant messaging messages, database requests, file system requests, or any other type of messages sent to a computer. In some embodiments, a message is received from a message generator such as a smart phone, a tablet device, a desktop computer, or any other network-enabled computing device. Then, routine 1300 proceeds to operation 1304 from operation 1302.

[0055] At operation 1304, the message processing engine 104 processes one or more messages from the incoming queue. In some embodiments, each message is processed by one of a set of processing threads, processors, services, or types of processing devices (e.g., CPU112A-112D). In some embodiments, processing a message from the incoming queue includes accepting the message. Accepting the message may include explicitly or implicitly indicating to the calling process that the message has been received and will be processed. In some embodiments, accepting the message may include adding the accepted message to the accepted queue 110. In some embodiments, the calling process (e.g., message builder 102) may expect to process the accepted message. In some embodiments, a high priority is given to the accepted queue, or the accepted queue is otherwise preferred.

[0056] The routine 1300 then proceeds to operation 1306 where the message processing engine 104 processes one or more messages from the accepted queue 110. In some embodiments, the same set of processing devices used to process messages from the incoming queue 108 is used to process messages from the accepted queue 110.

[0057] Then, the routine 1300 proceeds to operation 1308, where the processing message engine 104 allocates resources between the incoming queue 108 and the accepted queue 110. In some embodiments, computing resources are allocated based on the relative number of messages in each queue. However, in some embodiments, in order to avoid overwhelming the accepted queue 110, processing resources can be biased toward the accepted queue 110.

[0058] The routine 1300 then proceeds to operation 1310, where the message processing engine 104 determines whether the number of messages in the accepted queue meets or exceeds a specified threshold. Determining whether the number of messages in the accepted queue meets or exceeds a specified threshold can determine whether the server computing device 103 is approaching its capacity. Additionally or alternatively, determining that the number of messages in the accepted queue meets or exceeds a specified threshold can indicate that message processing delays are too high.

[0059] In some embodiments, the threshold may include an absolute number or a percentage of the maximum queue size. In some embodiments, the threshold may be dynamically determined based on the measured throughput of the message processing engine. For example, the message processing engine may try different thresholds and measure the final level of throughput (e.g., the number of messages processed per second), and then select the threshold that produces the highest throughput. In some embodiments, the threshold may be dynamically selected to minimize message processing delays (e.g., by trying different thresholds, measuring the average message throughput, and selecting the threshold that produces the lowest delay). In some embodiments, the threshold may be selected based on a combination of these and other factors, balancing user-defined preferences for low latency, high throughput, or other metrics (e.g., energy efficiency, tendency to crash, cooling metrics, or other aspects of operating the server computing device 103).

[0060] The routine 1300 then proceeds to operation 1312, where the message processing engine 104, in response to determining that the number of messages in the receive queue exceeds a specified threshold, blocks subsequent messages from being added to the incoming queue 108. By blocking messages from being added to the incoming queue 108, but allowing messages to continue to be added to the accepted queue 110, the message processing engine 104 creates breathing room while allowing existing messages to be processed uninterruptedly.

[0061] In some embodiments, messages are prevented from being added to the incoming queue 108 by discarding the messages (ie, not responding to them). In other embodiments, the message processing engine 104 may return an error code, for example, an indication that the server is full and temporarily unable to accept new processing messages.

[0062] Then, routine 1300 proceeds to operation 1314, where at operation 1314, message processing engine 104 transfers computing resources to process incoming queue 108. In some embodiments, all or substantially all computing resources are allocated to process messages from incoming queue 108, leaving almost no resources to process messages from accepted queue 110. In this way, the messages are in the incoming queue. However, in other embodiments, resources may be allocated between incoming queue 108 and accepted queue 110 in proportion to the number of messages stored in each queue, with or without a preference for incoming queue 108.

[0063] The routine 1300 then proceeds to operation 1316, where the message processing engine 104 transfers computing resources to process the accepted queue 110. In some embodiments, all or substantially all computing resources are transferred to process the accepted queue 110 so that the messages in the accepted queue 110 are processed as quickly as possible.

[0064] Then, routine 1300 proceeds to operation 1318, where at operation 1318, the message processing engine 104 determines whether the number of messages stored in the received queue 110 has dropped below a second threshold. In some embodiments, the second threshold is defined as an absolute number of messages, a percentage of the first threshold (e.g., 505 of the first threshold), etc. Additionally or alternatively, the second threshold can be dynamically defined based on feedback from the message processing engine. For example, when different second thresholds are adopted, overall throughput, message latency, and other indicators can be evaluated, and one or a combination of these factors can be used to select a second specified threshold. Routine 1300 proceeds from operation 1318 to operation 1320, where it ends.

[0065] It should be understood that the subject matter described above can be implemented as a computer-controlled device, a computer process, a computing system, or as an article such as a computer-readable storage medium. The operations of the example methods are shown in individual boxes and are summarized with reference to these boxes. These methods are shown as a logical flow of boxes, each box representing one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, these operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the described operations.

[0066] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be construed as limiting, and any number of the described operations may be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described processing. The described processing may be performed by resources associated with one or more devices (e.g., one or more internal or external CPUs or GPUs) and / or one or more hardware logic (e.g., a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other type of accelerator).

[0067] All of the methods and processes described above may be embodied and fully automated by software code modules executed by one or more general-purpose computers or processors. These code modules may be stored in any type of computer-readable storage medium or other computer storage device (e.g., those described below). Some or all of these methods may alternatively be embodied in dedicated computer hardware (e.g., those described below).

[0068] Any routine description, element or box in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or part of code, which includes one or more executable instructions for implementing a specific logical function or element in a routine. Alternative implementations are included within the scope of the examples described herein, where elements or functions may be deleted, or may be performed in a different order than shown or discussed, including substantially synchronously or in reverse order, depending on the functionality involved, as understood by those of ordinary skill in the art.

[0069] Fig.14 1400 for a computer capable of executing the program components described herein (e.g., a server computer 103 executing a message processing engine 104). Fig.14 The computer architecture 1400 shown in illustrative embodiments shows an architecture for a server computer, a mobile phone, a PDA, a smart phone, a desktop computer, a netbook computer, a tablet computer, and / or a laptop computer. The computer architecture 1400 can be utilized to perform any aspect of the software components presented herein.

[0070] Fig.14The illustrated computer architecture 1400 includes a central processing unit 1402 ("CPU"), a system memory 1404 (wherein the system memory 1404 includes a random access memory 1406 ("RAM") and a read-only memory ("ROM") 1408), and a system bus 1410 that couples the memory 1404 to the CPU 1402. A basic input / output system containing the basic routines that help to transfer information between various elements in the computer architecture 1400 (e.g., during startup) is stored in the ROM 1408. The computer architecture 1400 also includes a mass storage device 1412 for storing an operating system 1407, other data, and one or more message processing engines 104.

[0071] The mass storage device 1412 is connected to the CPU 1402 through a mass storage controller (not shown) connected to the bus 1410. The mass storage device 1412 and its associated computer readable media provide non-volatile storage for the computer architecture 1400. Although the description of computer readable media contained herein refers to a mass storage device (e.g., a solid state drive, a hard disk, or a CD-ROM drive), it should be understood by those of ordinary skill in the art that the computer readable media can be any available computer storage media or communication media that can be accessed by the computer architecture 1400.

[0072] Communication media includes computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transport mechanism), and includes any transmission media. The term "modulated data signal" means a signal whose one or more characteristics are changed or set in a manner that encodes information in the signal. By way of example, but not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic waves, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer readable media.

[0073] By way of example, and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of such data as computer-readable instructions, data structures, program modules, or other data. For example, computer media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disk ("DVD"), HD-DVD, BLU-RAY or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computer architecture 1400. For the purpose of convenience in describing the claims, the phrases "computer storage medium", "computer-readable storage medium" and variations thereof do not, by themselves, include waveforms, signals and / or other temporary and / or intangible communication media.

[0074] According to various configurations, the computer architecture 1400 can operate in a networked environment using logical connections to remote computers through the network 1456 and / or another network (not shown). The computer architecture 1400 can be connected to the network 1456 via a network interface unit 1414 connected to the bus 1410. It should be understood that the network interface unit 1414 can also be used to connect to other types of networks and remote computer systems. The computer architecture 1400 can also include an input / output controller 1416 for receiving and processing inputs from a computer system including a keyboard, a mouse, or an electronic stylus ( Fig.14 Similarly, the input / output controller 1416 can provide input to a display screen, a printer, or other types of output devices (not shown). Fig.14 ) to provide output.

[0075] It should be understood that the software components described herein can be loaded into CPU 1402 and executed, and CPU 1402 and the entire computer architecture 1400 can be converted from a general-purpose computing system to a special-purpose computing system customized to facilitate the functions presented herein. CPU 1402 can be composed of any number of transistors or other discrete circuit elements, which can present any number of states individually or collectively. Specifically, CPU 1402 can operate as a finite state machine in response to executable instructions contained in the software modules disclosed herein. These computer executable instructions can transform CPU 1402 by specifying how CPU 1402 switches between states, thereby transforming transistors or other discrete hardware elements that constitute CPU 1402.

[0076] Encoding the software modules provided herein can also transform the physical structure of the computer-readable medium provided herein. In different embodiments of this specification, the specific transformation of the physical structure can depend on various factors. Examples of these factors may include, but are not limited to: the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as a primary memory or an auxiliary memory, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of the transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also transform the physical state of these components to store data thereon.

[0077] For another example, the computer readable medium disclosed herein can be implemented using magnetic technology or optical technology. In such an implementation, when the software provided herein is encoded in a magnetic or optical medium, the software can transform the physical state of the magnetic or optical medium. These transformations can include changing the magnetic properties of a specific location in a given magnetic medium. These transformations can also include changing the physical characteristics or properties of a specific location in a given optical medium to change the optical properties of those locations. Without departing from the scope of protection and spirit of this specification, other transformations of physical media are also possible, and the aforementioned examples are just helpful to promote this discussion.

[0078] Based on the above, it should be understood that many types of physical transformations occur in the computer architecture 1400 in order to store and execute the software components presented herein. It should also be understood that the computer architecture 1400 may include other types of computing devices, including handheld computers, embedded computer systems, personal digital assistants, and other types of computing devices known to those of ordinary skill in the art. It is also contemplated that the computer architecture 1400 does not include Fig.14 All components shown in the Fig.14 Other components explicitly shown in the Fig.14 A completely different architecture than the one shown in .

[0079] Example clauses

[0080] The disclosure presented herein covers the subject matter set forth in the following example clauses.

[0081] Example 1: A system includes: one or more data processing units (1402); a computer readable medium (1404) having computer executable instructions (104) encoded thereon to cause the one or more data processing units (1402) to: receive a plurality of messages (113) into an incoming queue (108); add at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); determine a plurality of messages (113) in the accepted queue (110); The method of claim 100 wherein the number of messages in the received queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the received queue (110) exceeds the threshold number (120): blocking (302) subsequent messages from entering the incoming queue (108); transferring computing resources (112) to process the messages in the incoming queue (108); in response to the messages from the incoming queue (108) being processed, allocating computing resources (112) to process the messages from the received queue (110); and removing the blocking (302) to allow messages to enter the incoming queue (108).

[0082] Example 2: The system of Example 1, wherein messages from the accepted queue (110) are processed until the accepted queue (110) is exhausted.

[0083] Example 3: The system of Example 1, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0084] Example 4: The system of Example 1, wherein messages are allowed to be added to the accepted queue (110) while subsequent messages from the incoming queue (108) are blocked.

[0085] Example 5. A system according to Example 1, wherein adding at least one message (115) of the plurality of messages to the accepted queue (110) includes using the shared computing resources (112) to process messages (114) stored in the incoming queue (108).

[0086] Example 6: A system according to Example 1, wherein the computer executable instructions (104) further cause the one or more data processing units (1402) to: determine when the number of messages in the incoming queue (108) exceeds an incoming queue threshold (118); in response to determining that the number of messages in the incoming queue (108) exceeds the incoming queue threshold (118): prevent (302) subsequent messages from entering the incoming queue (108); determine whether the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue; and when the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110), bias computing resources toward the incoming queue (108).

[0087] Example 7: A system according to Example 1, wherein each message is associated with a customer account, wherein the customer account has a quota of pending input / output (IO) operations, and wherein messages from customers that have exceeded the quota of pending IO are prohibited from adding messages to the incoming queue (108).

[0088] Example 8: The system of Example 1, wherein the quota of pending IO operations is calculated based on an estimated amount of IO to complete messages in the incoming queue (108) and the accepted queue (110).

[0089] Example 9: The system of Example 1, wherein when the incoming queue (108) and the accepted queue (110) do not exceed their respective thresholds, the shared computing resources are allocated in proportion to the number of messages in each queue.

[0090] Example 10: The system of Example 9, wherein the shared computing resources are biased toward the accepted queue (110).

[0091] Example 11: A system comprising: one or more data processing units (1402); a computer readable medium (1404) having computer executable instructions (104) encoded thereon to cause the one or more data processing units (1402) to: receive a plurality of messages (113) into an incoming queue (108); add at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); ); determining that the number of messages in the accepted queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the accepted queue (110) exceeds the threshold number (120): blocking (302) subsequent messages from entering the incoming queue (108); in response to the messages from the incoming queue (108) being processed, allocating computing resources (112) to process the messages from the accepted queue (110); removing the block (302) to allow messages to enter the incoming queue (108).

[0092] Example 12: The system of Example 11, wherein messages from the accepted queue (110) are processed until the accepted queue (110) is exhausted.

[0093] Example 13: The system of Example 11, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0094] Example 14: The system of Example 11, wherein messages are allowed to be added to the accepted queue (110) while subsequent messages from the incoming queue (108) are blocked.

[0095] Example 15: A method employed by a computing device, comprising: receiving a plurality of messages (113) into an incoming queue (108); adding at least one message (115) of the plurality of messages to an accepted queue (110), wherein the incoming queue (108) and the accepted queue (110) share computing resources (112); determining that the number of messages in the accepted queue (110) exceeds a threshold number (120); in response to determining that the number of messages in the accepted queue (110) exceeds the threshold number (120), for: blocking (302) subsequent messages from entering the incoming queue (108); in response to the message from the incoming queue (108) being processed, allocating computing resources (112) to process the message from the accepted queue (110); and removing the block (302) to allow the message to enter the incoming queue (108).

[0096] Example 16: The method of Example 15 further includes: detecting when the computing resource 112 is overwhelmed, and in response to detecting that the computing resource 112 is overwhelmed, stopping processing messages from the incoming queue (108) and the accepted queue (110).

[0097] Example 17: The method according to Example 15 also includes: determining when the number of messages in the incoming queue (108) exceeds the incoming queue threshold (118); in response to determining that the number of messages in the incoming queue (108) exceeds the incoming queue threshold (118), for: blocking (302) subsequent messages from entering the incoming queue (108); determining whether the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110); and when the number of messages in the incoming queue (108) significantly exceeds the number of messages in the accepted queue (110), biasing computing resources toward the incoming queue (108).

[0098] Example 18: A method according to Example 15, wherein each message is associated with a customer account, wherein the customer account has a quota of pending input / output (IO) operations, and wherein messages from customers that have exceeded the quota of pending IO are prohibited from adding messages to the incoming queue (108).

[0099] Example 19: The method of Example 15, wherein the quota of pending IO operations is calculated based on an estimated amount of IO to complete messages in the incoming queue (108) and the accepted queue (110).

[0100] Example 20: The method of Example 15, wherein messages from the accepted queue (110) are processed until the number of messages in the requested queue (110) drops below a second threshold (602).

[0101] Among many other technical advantages, the techniques herein can more efficiently utilize computing resources such as processor cycles, memory, network bandwidth, and power consumption than previous solutions that flood server computers with messages. Another technical advantage is reduced message processing delays and avoidance of message timeouts when servers are overloaded. Other technical benefits not specifically mentioned herein can also be realized through implementations of the disclosed subject matter.

[0102] Although these techniques are described using language specific to structural features and / or methodological acts, it should be understood that the appended claims are not necessarily limited to these specific features or acts described above. Instead, these features and acts are merely described as exemplary embodiments of these techniques.

Claims

1. A system for managing messages by using an asymmetric cooperative queue, comprising: one or more data processing units; as well as A computer readable medium having computer executable instructions encoded thereon to cause the one or more data processing units to: Receive multiple messages into the incoming queue; adding at least one message of the plurality of messages in the incoming queue to the accepted queue, wherein shared computing resources are allocated between the incoming queue and the accepted queue; Determining that the number of messages in the received queue exceeds a threshold number; In response to determining that the number of messages in the accepted queue exceeds the threshold number: preventing subsequent messages from entering the incoming queue; diverting computing resources from the shared computing resources to process messages in the incoming queue; In response to the message from the incoming queue being processed, allocating computing resources from the shared computing resources to process the message from the accepted queue; and The block is removed to allow messages to enter the incoming queue.

2. The system according to claim 1, wherein: Process messages from the accepted queue until the accepted queue is exhausted.

3. The system according to claim 1, wherein: Messages from the accepted queue are processed until the number of messages in the requested queue drops below a second threshold.

4. The system according to claim 1, wherein: Messages are allowed to be added to the accepted queue while subsequent messages from the incoming queue are blocked.

5. The system according to claim 1, wherein: Adding at least one message of the plurality of messages to the accepted queue includes processing the messages stored in the incoming queue using the shared computing resources.

6. The system according to claim 1, wherein: The computer executable instructions further cause the one or more data processing units to: Determine when the number of messages in the incoming queue exceeds the incoming queue threshold: In response to determining that the number of the messages in the incoming queue exceeds the incoming queue threshold: preventing subsequent messages from entering the incoming queue; determining whether the number of the messages in the incoming queue significantly exceeds the number of the messages in the accepted queue; and When the number of the messages in the incoming queue significantly exceeds the number of the messages in the accepted queue, computing resources are biased toward the incoming queue.

7. The system according to claim 1, wherein: Each message is associated with a customer account, wherein the customer account has a quota of pending input / output (IO) operations, and wherein messages from customers that have exceeded the quota of pending IO are prohibited from having messages added to the incoming queue.

8. The system according to claim 7, wherein: The quota of pending IO operations is calculated based on the estimated amount of IO to complete the messages in the incoming queue and the accepted queue.

9. The system according to claim 1, wherein: When the incoming queue and the accepted queue do not exceed their respective thresholds, the shared computing resources are allocated in proportion to the number of messages in each queue.

10. The system according to claim 9, wherein: Shared computing resources are biased towards the accepted queues.

11. A system for managing messages by using an asymmetric cooperative queue, comprising: one or more data processing units; as well as A computer readable medium having computer executable instructions encoded thereon to cause the one or more data processing units to: Receive multiple messages into the incoming queue; adding at least one message of the plurality of messages in the incoming queue to the accepted queue, wherein shared computing resources are allocated between the incoming queue and the accepted queue; Determining that the number of messages in the received queue exceeds a threshold number; In response to determining that the number of messages in the accepted queue exceeds the threshold number: preventing subsequent messages from entering the incoming queue; In response to the message from the incoming queue being processed, allocating computing resources from the shared computing resources to process the message from the accepted queue; and The block is removed to allow messages to enter the incoming queue.

12. A method for managing messages by using an asymmetric cooperative queue, comprising: Receive multiple messages into the incoming queue; adding at least one message of the plurality of messages in the incoming queue to the accepted queue, wherein shared computing resources are allocated between the incoming queue and the accepted queue; Determining that the number of messages in the received queue exceeds a threshold number; In response to determining that the number of messages in the accepted queue exceeds the threshold number: preventing subsequent messages from entering the incoming queue; In response to the message from the incoming queue being processed, allocating computing resources from the shared computing resources to process the message from the accepted queue; and The block is removed to allow messages to enter the incoming queue.

13. The method according to claim 12, further comprising: Detecting when a computing resource is overwhelmed, and in response to detecting that the computing resource is overwhelmed, stopping processing messages from both the incoming queue and the accepted queue.

14. The method according to claim 12, further comprising: Determine when the number of messages in the incoming queue exceeds the incoming queue threshold: In response to determining that the number of the messages in the incoming queue exceeds the incoming queue threshold: preventing subsequent messages from entering the incoming queue; determining whether the number of the messages in the incoming queue significantly exceeds the number of the messages in the accepted queue; and When the number of the messages in the incoming queue significantly exceeds the number of the messages in the accepted queue, computing resources are biased toward the incoming queue.

15. The method according to claim 12, wherein: Messages from the accepted queue are processed until the number of messages in the requested queue drops below a second threshold.

Images