Data service overload detection and mitigation

Abstract

The described techniques are intended to present a server overload condition that may be caused by a surge in client requests. Upon receiving an incoming client request for data at a server, a request buffer status (e.g., a counter) of a client request buffer is evaluated to determine whether the client request corresponds to an expected overload condition with respect to the request processing capabilities of the server, e.g., whether the client request buffer is full. If so, an overload response (e.g., a retry response) is returned to the client to instruct the client to retry the corresponding client request after a time delay. If not in the expected overload condition, the request is further processed, e.g., by adding a request information entry associated with the incoming client request to the client request buffer and updating the request buffer status, to, for example, provide a regular response containing the requested data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 16 / 009,729, filed on June 15, 2018, entitled “DATA SERVICE OVERLOAD DETECTION AND MITIGATION,” the entire contents of which are incorporated herein by reference. Background Art

[0003] Many contemporary computing device programs have online components that make requests to external data services as online clients. In order to scale up to handle a large number of such client requests, a typical data service solution is to receive client requests at a load balancer, which distributes the client requests among many possible data service servers, usually referred to as server instances. The servers then process and send responses to the client requests.

[0004] One of the operations performed by the data service server is to respond to periodic heartbeat detection requests from the load balancer, for example, sent through a specific URL understood by the load balancer and the server for heartbeat communication. When no heartbeat from the data service server is detected, the data service server is removed and restarted.

[0005] Data services encounter problems when large, sudden loads of client requests occur, sometimes at unpredictable times. Often, such surges in demand overload data service servers as they try to keep up with the many client requests for data. When overloaded, data service servers often fail to respond to heartbeat checks, even though the service is normally operational, causing them to reboot. Taking down servers for reboot operations further stresses other overloaded servers. Summary of the invention

[0006] The present invention summary is provided to introduce a series of representative concepts in a simplified form, which are further described in the following detailed description. The present invention summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way to limit the scope of the claimed subject matter.

[0007] Briefly, various aspects of the technology described herein are directed to avoiding a server overload condition. Upon receiving an incoming client request for data at a server, various aspects include: evaluating a request buffer status of a client request buffer to determine whether the client request corresponds to an expected overload state with respect to the request processing capabilities of the server. If evaluating the request buffer status determines that the incoming client request corresponds to the expected overload state, then described herein is returning a response to the client indicating that the client is to retry the corresponding client request after a time delay. If evaluating the request buffer status determines that the incoming client request does not correspond to the expected overload state, then various aspects include adding a request information entry associated with the incoming client request to the client request buffer, and updating the request buffer status.

[0008] Other advantages may become apparent from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The technology described herein is illustrated by way of example and not limitation by the accompanying drawings, in which like reference numerals indicate like elements, and in which:

[0010] Figure 1 is a block diagram representing example components including processing a client request for data via a server having overload detection capabilities according to one or more example implementations.

[0011] Figure 2 is a block diagram representing example components of a client and server with overload detection capabilities in which an overload condition is not currently anticipated according to one or more example implementations.

[0012] Figure 3 is a block diagram representing example components of a client and server with overload detection capabilities in which an expected overload condition is detected according to one or more example implementations.

[0013] Figure 4 is a flowchart representing example logic / operations that may be performed by a server regarding detecting whether an expected overload condition exists and subsequent operations according to one or more example implementations.

[0014] Figure 5 is a flow diagram representing example logic / operations that may be performed by a server regarding processing client requests for data while tracking possible overload-related conditions, according to one or more example implementations.

[0015] Figure 6is a flow diagram representing example logic / operations that may be performed by a client regarding sending a client request for data and processing a response corresponding to an expected overload condition of a server according to one or more example implementations.

[0016] Figure 7 is a flow diagram illustrating example logic / operations that may be performed to return an expected overload condition response according to one or more example implementations.

[0017] Figure 8 is a block diagram illustrating example logic for avoiding server overload when processing client requests according to one or more example implementations.

[0018] Fig. 9 is a flow diagram illustrating example logic / operations that may be performed to process client requests or return an expected overload condition response based on client request buffer conditions according to one or more example implementations.

[0019] Fig.10 is a block diagram representing an example computing environment that can be incorporated into aspects of the subject matter described herein. DETAILED DESCRIPTION

[0020] Aspects of the technology described herein are generally directed to avoiding overloading a server due to client data requests. In general and as described herein, before buffering a client data request for processing, the condition of the buffer is evaluated to determine whether subsequent processing of the client request will result in an expected overload condition. If subsequent processing of the client request will not result in an expected overload condition, the request is buffered, such as for conventional processing, otherwise a response to the request is returned to indicate the expected overload condition. The response may indicate that the corresponding request is to be retried after a delay time; in one or more implementations, the delay time may be specified in the response.

[0021] In one or more aspects, the condition of the buffer is tracked via an overload counter that maintains information about how full the buffer is (e.g., a simple entry count). In one or more implementations, the overload counter is incremented when a new client request entry is placed into the buffer, and the overload counter is decremented when a buffered client request entry is removed from the buffer for processing. When an incoming client request is received, the overload counter is compared to a threshold that determines whether the server is in an expected overload state. In this manner, the client request buffer does not become overfilled and cause actual server overload.

[0022] It should be understood that any examples herein are non-restrictive. For example, some examples relate to HTTP: (Hypertext Transfer Protocol) requests and responses, however, the technology is not limited to any request / response protocol. As another example, the technology is illustrated as being implemented in a data service server serving a remote client, however, the technology described herein can be implemented in a single server, or in any entity providing data that may be overloaded by requests including local requests. Therefore, the technology described herein is not limited to any specific implementation, aspect, concept, structure, function, or example described herein. On the contrary, any implementation, aspect, concept, structure, function, or example described herein is non-restrictive, and the technology can be used in various ways, which generally provide benefits and advantages in computing and data transmission.

[0023] Figure 1 is a block diagram representing example components that may be used to provide aspects of and / or work in conjunction with the techniques described herein. Figure 1 , a request from client 102 to data service 104 is received at load balancer 106, (which may be considered part of data service 104). As is known, load balancer 106 distributes the request to servers 108(1) to 108(n) of data service 104, for example, via a suitable load balancing technique such as round-robin distribution.

[0024] As described herein, the example servers 108(1)-108(n) include overload detection logic 110(1)-110(n) that detects when the server is in an "expected overload" state, where "expected overload" as used herein means that the server is still functioning, however any additional load on the server may cause the server to fail to properly respond to health check / heartbeat (HB) checks, e.g. Figure 1 As shown by the oblique dashed line marked as HB in the figure. As is known, when a server instance becomes overloaded, as the server instance uses more and more resources, the latency of the server instance response increases, and eventually the server instance fails to comply with the health check / heartbeat detection in a timely manner. Failure to comply with the health check / heartbeat detection causes the health check / heartbeat detection logic 112 of the load balancer 106 (the health-related components contained therein) to restart the server instance. Note that restarting the server instance will distribute the load among the remaining servers, which may cause one or more of the remaining servers to fail the health check / heartbeat compliance response, and so on.

[0025] As described herein, overload detection logic (e.g., 110(1)) detects an expected overload condition and takes action to avoid an actual overload that would result in a server failure. In one or more aspects, overload detection logic 110(1) returns a response to a client request (using relatively few computing resources) indicating that the client request is to be retried after a delay time. The response may indicate a delay time, or the client may use a default delay time.

[0026] As an example, in an HTTP: protocol environment, when an expected overload state is detected at the overload detection logic 110(1) of the server, a '509' response that may specify a delay time may be returned directly in response to a client request. Note that this is in contrast to a conventional '503' service unavailable response, which would cause the client to immediately resend the request, possibly not giving the server enough time to reduce its load. In contrast, a '509' "retry after delay time" response gives the server time to process its outstanding requests, e.g., pending requests in a request buffer, etc. It should be noted that a '509' response is not an official HTTP response code, and that the client and server may therefore negotiate conventions on how to handle such responses, etc. For example, a service may send a '509' response with a header "x-retry-interval: 3000" to request that the client retry after 3 seconds. Thus, for example, the server may adjust the interval based on how many '509' responses have been recently issued, thereby controlling the number and / or timing of retries.

[0027] Figure 2 Various aspects of the overload detection and avoidance techniques described herein are shown from the perspective of client 202 and server (instance) 208. Figure 2 In the example of FIG. 1 , input 220 to a request / response program 222 (such as a user interface or UI of an application) as represented by the arrow labeled one (1) causes a request generator 226 to generate a request 224 (arrow labeled two (2)). For example, the request may be for a request from Figure 1 The data service 104 receives an "HTTP: GET" request for data, etc.

[0028] In this particular example, request 224 is distributed to server 208 via load balancer 106 (arrow three (3)). As described herein, overload detection logic 228 receives incoming request 224 and decides how to handle request 224 taking into account an expected overload condition. In one or more implementations, the overload detection logic evaluates the current request buffer condition, for example, by reading an overload counter 230 that tracks the current number of outstanding requests in request buffer 232 (arrow four (4)); (the overload counter can therefore be considered a request buffer counter). If this number is at a threshold, such as fifty, (which can be a configurable value based on any one or more criteria of the particular server instance, such as processing power, memory, etc.), then an expected overload condition is detected. However, in the event of an expected overload condition, the overload detection logic 228 may be used to detect the expected overload condition. Figure 2 In the example of , the expected overload state has not been reached, so the overload counter is incremented (arrow five (5)), and the request (e.g., an information entry associated with the request, which may be the request data itself) is placed in the request buffer (arrow six (6)). Note that the information entry associated with the request may be the request data itself, however, instead of copying the request data into the buffer, a pointer, index, etc. of the request data (e.g., already loaded in the memory) may be placed in the buffer as the information entry associated with the request.

[0029] like Figure 2 As represented by arrow seven (7) in the example, the request processing logic 234 processes the outstanding request to return an appropriate response 236 to the client (arrow twelve (12)), including a response 236j to the client 202 corresponding to the request 224 (arrow thirteen (13)). Before returning the response, in this example, the request processing logic 234 removes the request information entry from the request buffer 232 and, after reading the overload counter 230 to ensure that it is not already at zero (arrow eight (8)), decrements the overload counter 230 (arrow eight (9)). Note that the response may include data obtained from the external data storage device 238 (arrow ten (10) and arrow eleven (11)), however, instead of or in addition to the external data, internal data, cached data, etc. may be the source or partial source of the returned data. It should also be noted that the response may be an error response code, etc. (but typically not an error related to an overload that would normally be detected by the overload detection logic 228).

[0030] exist Figure 2In the example of , the response processing logic 240 of the client 202 recognizes that the response is for the program 222, where the response is sent to be processed by the data response processing logic 242 (arrow fourteen (14)), for example, to render the output 244 (arrow fifteen (15)). Note that in some cases, the data response processing logic 242 can communicate with the request generator 226 (arrow sixteen (16)) to retrieve related / additional data, for example, based on the returned response data.

[0031] In this manner, the client and server operate normally until request buffer 232 becomes overfull, as detected by overload counter 230 and evaluated against a threshold. Figure 2 Heartbeat response logic 246 (slanted dashed line labeled HB) is shown complying with the health check / heartbeat requirements. More generally, a server communicates with a component of a data service (eg, a load balancer) to provide health-related information indicating that the server is currently functioning.

[0032] Figure 3 Various aspects of the overload detection and avoidance techniques described herein are shown from the perspective of client 202 and server (instance) 208; however, in Figure 3 In , server instance 208 is in an expected overload state. Figure 3 In the example of FIG. 1 , another input 320 to the request / response program 222 (such as a user interface or UI of an application), as represented by the arrow labeled twenty-one (21), causes the request generator 226 to generate another request 324 (arrow labeled twenty-two (22)). For example, the request may be for a request from Figure 1 The data service 104 receives an "HTTP: GET" request for data, etc.

[0033] In this particular example, the other request 324 is distributed to the server 208 by the load balancer 106 (arrow twenty-three (23)). As described herein, the overload detection logic 228 receives the request 324 and considers the expected overload condition to decide how to handle the request 324. In one or more implementations, as described above, the overload detection logic reads the overload counter 230 (arrow twenty-four (24)), which tracks the current number of outstanding requests in the request buffer 232. If the number is at a threshold, an expected overload condition is detected.

[0034] and Figure 2 In contrast to the example of Figure 3 In the example above, the expected overload condition has been reached. Figure 2 Different, in Figure 3In the illustrated implementation, the overload counter 230 is not incremented and the request is not placed in the request buffer 232.

[0035] Instead, overload detection logic 228 generates (or causes to be generated) a "retry after delay" response 348k to indicate that client 202 should retry the request after a certain delay time (arrow twenty-five (25)); the delay interval may be included in the response. Figure 3 In the example of , the response is an HTTP: 509 response and includes a delay time equal to X, such as 1000 milliseconds. In this expected overload state, the overload detection logic 228 can also send a "retry after delay" response (collectively marked as 348) to other clients regarding any other requests sent by the client 202.

[0036] Note that request processing logic 234 continues to process outstanding requests pending in request buffer 232 to return an appropriate response 336 to the client. This may or may not include responses to some other requests previously made by client 202, i.e., responses that do not correspond to request 324 at this time. In this way, while client 202 (and possibly other clients) waits to resend requests, request buffer 232 is drained and the overload counter is decremented.

[0037] exist Figure 3 In the example of , the response processing logic 240 of the client 202 receives the response 348k (arrow twenty-six ((26)), recognizes that the response is for the program 222, but the response 348k is an overload-related response. In this example, the response 348k is sent to be processed by the overload detection processing logic 248 (arrow twenty-seven ((27)), for example, to be delayed, and then communicated with the request generator 226 (arrow twenty-eight ((28)) or communicated with some other logic of the program 222, for example, to regenerate a request corresponding to the original request.

[0038] In this manner, when request buffer 232 becomes overfilled, the server stops processing requests for data (although it will be appreciated that new requests may be buffered for processing at the rate at which pending requests are processed). Instead, client requests are throttled using only a relatively small amount of resources via a "try again after a delay" response. As a result of avoiding actual overload, heartbeat response logic 246 is able to continue to honor health check / heartbeat requirements (the diagonal dashed line marked HB), and the server continues to respond to as many pending requests as possible as quickly as possible without being restarted by the load balancer (assuming no other failures occur).

[0039] Figure 4 It is shown that it can be obtained by, for example Figure 2 and Figure 3Flowchart of example operations taken by the server overload detection logic of the logic 228 of . Operation 402 represents receiving a request at a server, at which point the request arrives at the server overload detection logic. Operation 404 reads the overload counter, and operation 406 evaluates whether the overload counter is at a threshold limit. If not at the limit, operation 408 increments (or otherwise increases / modifies) the overload counter, and operation 410 adds the associated request information entry to the pending request buffer, as generally described herein. Note that operations 408 and 410 can be performed in a transactional manner as critical section / mutex operations, etc., so that when, for example, request processing logic operating in parallel is removing request items from the buffer and decrementing the overload counter, the count remains correct.

[0040] If, on the contrary, operation 406 evaluates that the limit has been reached, then operation 412 discards the request, which may or may not delete the request from the server. Note that "discarding" may, for example, simply mark the request for discarding (e.g., via later garbage collection), because some or all requests may be retained in the knowledge that retries after delays may occur, and / or any processing results of the request may be retained. For example, (at the expense of a relatively small amount of storage), the server may use fewer processing resources when processing a duplicate request that is identified as corresponding to a request that was marked for discard but still exists to some extent in memory.

[0041] Operation 414 returns a "retry after delay" response, such as an HTTP: 509 response, which may include a delay time. Typically, most responses may be preloaded into memory and copied from memory to provide efficient response generation when needed. Note that some hysteresis or similar mechanisms may be used, such as triggering an HTTP: 509 response once the overload counter reaches fifty, and continuing to do so until the overload counter / request buffer is reduced to forty-five.

[0042] It can be seen that Figure 4 The operation can be implemented using relatively few computing resources. Note also that, although not shown, a mechanism such as another counter can be used to track, for example, how many HTTP: 509 responses have been sent in a given time period and / or regarding successfully processed requests, etc., to assist the operator of the data service in the subsequent allocation of resources.

[0043] Figure 5Example operations of the data request processing logic of the server when a request from a buffer is to be processed are shown, starting at operation 502 where an overload counter is read. Operation 504 represents a check to see if the overload counter is already at zero, and if not, operation 506 decrements the overload counter. Note that operation 504 may be unnecessary if the counter value is guaranteed to be correct, for example, via critical section / mutex protection, and thus, as represented herein, operation 504 may simply be a safety check to ensure that the counter never becomes negative. It should also be noted that counting in the opposite direction is essentially the same concept, e.g., starting a "buffer count remaining" counter at a threshold, and decrementing such a counter when items are added to the buffer, and incrementing the counter when items are removed from the buffer.

[0044] As can be readily appreciated, maintaining a counter is only a relatively efficient way to determine whether a buffer is full. Other ways may include evaluating the difference between a buffer start pointer and an end pointer, and so on.

[0045] Operation 508 removes the request information entry from the buffer, which, as described herein, may be the associated request data itself, or a pointer (or index, etc.) to the request data, in which case operation 508 also accesses the request data. Note again that operations 506 and 508 (and possibly operation 504 as well) may need to be performed as a critical section to ensure that the count remains correct, since the overload detection process may otherwise increment the overload counter in parallel overall.

[0046] Operation 510 represents the data request processing logic processing the request, such as obtaining the requested data. Operation 512 represents generating a response and sending it to the client. Note that the response may include data, but alternatively, for example, if the data cannot be obtained, it may include an error.

[0047] It should be understood that Figure 4 and Figure 5 At least some of the server operations of and other flow charts described herein may be combined into a single process. Figure 4 and / or Figure 5 At least some operations of other flow charts described herein may be performed in parallel with other operations. In addition, it should be understood that the order of at least some operations may be different from the order in the example flow charts. Therefore, the present technology is not limited to the example components and / or example operations shown herein, nor is it limited to the separation and / or sequencing of their examples described herein.

[0048] Although Figure 4 and Figure 5 The example operation of provides a practical implementation method to avoid server overload, but it should be understood that Figure 4and Figure 5 The concept can be extended. For example, different request buffers can be used for different types of requests. For example, certain types of requests can be detected as being able to be processed with low computing resources, while other types of requests can correspond to resource-intensive requests. Overload detection can be configured to detect such different requests and classify them into, for example, different buffers, and each different type of request buffer can have a different threshold. In this way, requests that can be processed efficiently and quickly can have priority over requests that need to be processed with more time / resources.

[0049] Alternatively, a single buffer may be used for different types of requests. However, instead of incrementing a counter, a weight may be added to the counter that varies depending on the type of request. For example, request type A may have a weight of 1, while request type B may have a weight of 3, because, (as determined by, for example, testing), three requests of type A may be processed at the same time / using the same amount of computing resources as one request of type B. The overload counter threshold may be set to an appropriate value to avoid overloading the server, whether all types A, all types B, or a mix of types A and B fill the buffer to the allowed amount. In this way, a surge in (mostly) type A requests avoids sending HTTP:509 responses (at least for a period of time) based on the overload counter threshold, but is still able to avoid server overload, which is set conservatively as if all requests were of the same type to guard against the worst-case scenario of a surge in type B requests.

[0050] Figure 6 An example operation of a client is shown, beginning at operation 602 representing sending a request. Operation 604 represents waiting until a response associated with the request at operation 602 has been received; note that waiting is typically not a loop, but can be based on an event-driven concept.

[0051] Once a response is received, operation 606 evaluates whether the response is a "retry after delay" response as described herein, such as an HTTP: 509 response. If not, operation 606 branches to operation 608, which represents processing the associated response.

[0052] If in operation 606, the response is a "retry after delay" response, then operation 610 represents obtaining a delay time from the response. Note that if no delay time is provided, a default delay time may be used.

[0053] Operation 612 represents resetting and starting the delay timer. Operation 614 represents waiting until the time is reached, and can also be (and usually is) event-driven rather than an actual loop. When the time has been reached, operation 614 returns to operation 602 to resend the request. Although not shown, there may be some kind of retry limit counter, etc., to avoid infinite attempts without trying something else.

[0054] Note that in some cases it is feasible for the client to maintain other data requests during the delay time (e.g., of operation 614). For example, if the client has just received delayed retry responses to multiple near-simultaneous requests, the client may not send another similar request until after a certain delay (not necessarily the same amount of delay as in operation 614; such a different delay may be shorter, e.g., not waiting the full amount of time for some independent data). In addition, if the client knows or can determine that it often has its requests load balanced to the same server, the client may also delay based on the likelihood that the server is close to being overloaded. Note that in many cases, subsequent requests depend on data returned in responses to previous requests, so such delays may sometimes be inherent without any other latency considerations on the client side.

[0055] In general, the technology described herein solves the problem of unexpected surges in client requests that may occur. Typically, if a surge in client requests is expected, additional servers can be pre-allocated. For example, consider that a very large number of requests associated with a popular television program are known to arrive when the program is first broadcast at 9:00 p.m. Eastern Time; in such a case, additional servers are allocated to avoid disappointing or frustrating client viewers. In some unexpected surge scenarios, servers can be added as needed, however, it takes a considerable amount of time (e.g., about a few minutes) to start such additional servers. Therefore, the technology described herein is well adapted to the scenario of adding servers on demand, because the technology described herein avoids overloading existing servers (and causing additional unnecessary restarts) until additional servers can be started.

[0056] As can be seen, server overload can be avoided via the techniques described herein, in which an expected server overload state is detected and actions are taken to avoid an actual server overload condition. In the expected overload state, the server responds with a "retry after delay" delayed response, which is an efficient response relative to conventional response processing, while suppressing additional client requests. This technique gives the server time to reduce its pending requests without the conventional '503' service unavailable response, etc., which would cause the client to retry immediately, thereby delaying the surge in client requests.

[0057] One or more aspects of the technology described herein are directed to Figure 7 Example operation of . Figure 7 Operation 702 of the embodiment represents receiving a client request for information at a server. Operation 704 represents determining that the client request corresponds to an expected overload state with respect to the request processing capacity of the server. Operation 706 represents returning a response corresponding to the expected overload state to the client in response to determining the expected overload state.

[0058] Returning a response to the client corresponding to the expected overload status may include returning a retry response to instruct the client to retry the request after a time delay. Returning a response to the client corresponding to the expected overload status may include instructing the client to retry the request after a time delay, wherein the time delay is specified in the response. Returning the response to the client may include returning an HTTP: 509 response.

[0059] Determining that the client request corresponds to an expected overload state may include evaluating a request buffer counter.

[0060] One or more aspects may include discarding a client request for information.

[0061] The server may be part of a data service, and aspects may include components for transmitting health-related information from the server to the data service.

[0062] Figure 8800. An example component of a server 800 that can implement various aspects of the technology described herein is represented. Box 802 represents an overload detection logic component that processes incoming client requests from a client and determines whether subsequent processing of the incoming client requests will result in an expected overload condition with respect to the request processing capacity of the server. As represented by box 804, if it is determined that subsequent processing of the incoming client requests will not result in an expected overload condition, the overload detection logic component adds a request information entry associated with the incoming client request to a client request buffer (box 806), and increases a counter for tracking pending client requests in the client request buffer (box 808). If (box 810) it is determined that subsequent processing of the incoming client request will result in an expected overload condition, the overload detection logic component discards the incoming client request (box 812), and returns a retry response (box 814) indicating to the client that the client will retry the corresponding client request after a time delay. The server 800 also includes a request processing logic component (box 816), which removes the request information entry associated with the client request from the client request buffer. As represented by block 818, for the removed request information entry, the request processing logic component decrements the counter (box 822), processes the request data associated with the removed request information entry to obtain response data (box 822), and returns the response data in response to the client request corresponding to the removed request information entry (box 824).

[0063] The server can be part of a data service, and the server can be coupled to a health-related component of the data service to transmit health-related information to the health-related component of the data service; the health-related information can indicate that the server is currently operational within the data service. Aspects can include a load balancer of the data service coupled to the server, and receiving incoming client requests from clients at the server through the load balancer, and the health-related component of the data service can be incorporated into the load balancer.

[0064] Subsequent processing of the incoming client request may be determined to result in an expected overload condition, and the retry response may include time delay data specifying how long the client is to delay before retrying the corresponding client request. Subsequent processing of the incoming client request may be determined to result in an expected overload condition, and the retry response may include an HTTP: 509 response.

[0065] Fig. 9904 represents an example operation performed, for example, when executing a machine-readable storage medium having machine-executable instructions. Operation 902 represents receiving an incoming client request for data at a server. Operation 904 represents evaluating a request buffer status of a client request buffer to determine whether the client request corresponds to an expected overload state with respect to the request processing capacity of the server. If evaluating the request buffer status determines that the incoming client request corresponds to the expected overload state (box 906), then various aspects include returning a retry response to the client indicating that the client will retry the corresponding client request after a time delay (box 908). If evaluating the request buffer status determines that the incoming client request does not correspond to the expected overload state (box 910), then various aspects include adding a request information entry associated with the incoming client request to the client request buffer (box 912), and updating the request buffer status (box 914).

[0066] Returning a retry response to the client may include instructing the client to retry the request after a time delay, wherein the time delay is specified in the response. Returning a response to the client may include returning an HTTP:509 response.

[0067] Evaluating the request buffer status may include evaluating an overload counter that tracks the number of entries in the request buffer.Updating the request buffer status may include incrementing the overload counter in conjunction with adding a request information entry associated with the incoming client request to the client request buffer.

[0068] Aspects may include removing a request information entry associated with the client request from a client request buffer, and decrementing an overload counter for the removed request information entry.

[0069] Aspects may include processing request data associated with the removed request information entry to obtain response data, and returning the response data in response to the client request corresponding to the removed request information entry.

[0070] If evaluating the request buffer status determines that the incoming client request corresponds to an expected overload condition, other aspects may include discarding the client request.

[0071] Example computing device

[0072] The techniques described herein may be applied to any device or collection of devices (machine) capable of running programs and processes. Thus, it is understood that personal computers, laptops, handheld, portable and other computing devices, and various computing objects including cell phones, tablet / tablet computers, gaming / entertainment consoles, etc. are contemplated for use in conjunction with various implementations including those exemplified herein. Thus, below in Fig.10The general purpose computing mechanism described in is only one example of a computing device.

[0073] Implementations may be implemented in part via an operating system for use by developers of devices or services of an object, and / or included within application software that operates to perform one or more functional aspects of the various implementations described herein. Software may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers, such as client workstations, servers, or other devices. Those skilled in the art will recognize that computer systems have various configurations and protocols that may be used to communicate data, and therefore, no particular configuration or protocol is considered limiting.

[0074] therefore, Fig.10 An example of a suitable computing system environment 1000 is shown in which one or more aspects of the implementations described herein may be implemented, although as noted above, the computing system environment 1000 is merely one example of a suitable computing environment and is not intended to suggest any limitation as to scope of use or functionality. Additionally, the computing system environment 1000 is not intended to be interpreted as having any dependency relating to any one or combination of components illustrated in the example computing system environment 1000.

[0075] Reference Fig.10 , an example apparatus for implementing one or more implementations includes a general purpose computing device in the form of a computer 1010. Components of the computer 1010 may include, but are not limited to, a processing unit 1020, a system memory 1030, and a system bus 1022 that couples various system components including the system memory to the processing unit 1020.

[0076] Computer 1010 typically includes a variety of machine (e.g., computer) readable media and may be any available media that can be accessed by a machine such as computer 1010. System memory 1030 may include computer storage media in the form of volatile memory and / or non-volatile memory such as read-only memory (ROM) and / or random access memory (RAM), as well as hard drive media, optical storage media, flash memory media, etc. By way of example and not limitation, system memory 1030 may also include an operating system, application programs, other program modules, and program data.

[0077] A user may enter commands and information into the computer 1010 through one or more input devices 1040. A monitor or other type of display device is also connected to the system bus 1022 via an interface such as output interface 1050. In addition to the monitor, the computer may also include other peripheral output devices such as speakers and printers, which may be connected through output interface 1050.

[0078] The computer 1010 may operate in a networked or distributed environment using logical connections to one or more other remote computers, such as a remote computer 1070. The remote computer 1070 may be a personal computer, a server, a router, a network PC, a peer device or other public network node, or any other remote media consumption or transmission device, and may include any or all of the elements described above with respect to the computer 1010. Fig.10 The logical connections depicted in the diagram include a network 1072 such as a local area network (LAN) or a wide area network (WAN), but other networks / buses may also be included. Such networking environments are common in homes, offices, enterprise-wide computer networks, intranets, and the Internet.

[0079] As noted above, although example implementations have been described in conjunction with various computing devices and network architectures, the underlying concepts may be applied to any network system and any computing device or system suitable for implementing such techniques.

[0080] In addition, there are multiple ways to implement the same or similar functionality, e.g., appropriate APIs, toolkits, driver code, operating systems, controls, standalone or downloadable software objects, etc., that enable applications and services to take advantage of the techniques provided herein. Thus, the implementations herein are considered from the perspective of an API (or other software object) and from the perspective of a software or hardware object that implements one or more implementations as described herein. Thus, the various implementations described herein may have aspects that are entirely in hardware, partially in hardware and partially in software, and entirely in software.

[0081] The word "example" is used herein to mean used as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as an "example" is not necessarily to be interpreted as being more preferred or advantageous than other aspects or designs, nor is it meant to exclude equivalent example structures and techniques known to those of ordinary skill in the art. In addition, with respect to the use of the terms "including," "having," "comprising," and other similar words, for the avoidance of doubt, these terms are intended to be inclusive in a manner similar to the term "comprise" as an open transition word, and do not exclude any additional or other elements when used in the claims.

[0082] As mentioned above, the various techniques described herein can be implemented in combination with hardware or software or in combination with a combination of hardware and software where appropriate. As used herein, the terms "component", "module", "system", etc. are also intended to represent an entity related to a computer, a combination of hardware, hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and / or a computer running on a processor. For example, both an application and a computer running on a computer can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers.

[0083] The above-mentioned system is described about the interaction between several components.It is to be understood that such system and components can include some and / or other components in those components or specified subcomponents, specified components or subcomponents, and according to the aforementioned various permutations and combinations.Subcomponents can also be implemented as components that are communicatively coupled to other components, rather than being included in the parent component (grading).In addition, it can be noted that one or more components can be combined into a single component that provides comprehensive functions or be divided into several separated subcomponents, and any one or more intermediate layers such as management layers can be provided to be communicatively coupled to such subcomponents to provide integrated functions.Any components described herein can also interact with one or more other components that are not specifically described herein but are generally known to those skilled in the art.

[0084] In view of the example systems described herein, the methods that can be implemented according to the described subject matter can be understood with reference to the flowcharts / flowcharts of the various accompanying drawings. Although the methods are shown and described as a series of frames for the purpose of simplifying the description, it should be understood and appreciated that various implementations are not limited by the order of the frames, because some frames can occur in different orders and / or occur simultaneously with other frames depicted and described herein. In the case of non-sequential or branched processes shown via flow charts / flowcharts, it is understood that various other branches, process paths, and the order of frames that obtain the same or similar results can be implemented. In addition, some of the illustrated frames are optional when implementing the methods described herein.

[0085] in conclusion

[0086] Although the present invention is susceptible to various modifications and alternative constructions, some of its illustrated implementations are shown in the drawings and have been described in detail above. However, it should be understood that it is not intended to limit the present invention to the specific forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions and equivalents that fall within the spirit and scope of the present invention.

[0087] In addition to the various implementations described herein, it should be understood that other similar implementations may be used, or modifications and additions may be made to the described implementations without departing from the described implementations to perform the same or equivalent functions of the corresponding implementations. Further, multiple processing chips or more devices may share the performance of one or more functions described herein, and similarly, storage may be implemented on multiple devices. Therefore, the present invention is not limited to any single implementation, but should be interpreted in breadth, spirit, and scope according to the appended claims.

Claims

1. A method for data service overload detection and mitigation, comprising: At the server, receiving a client request for information; determining that the client request results in an expected overload condition with respect to request processing capacity of the server; wherein the expected overload state is defined by a buffer threshold at which the buffer of the server is not full; as well as In response to determining the expected overload condition, the client request for information is discarded and a retry response is returned to the client instructing the client to retry the client request after a time delay.

2. The method according to claim 1, wherein: Returning the retry response to the client includes returning an HTTP:509 response.

3. The method according to claim 1, wherein: The time delay is specified in the retry response.

4. The method according to claim 1, wherein: Determining that the client request results in the expected overload condition includes evaluating a request buffer counter and comparing the request buffer counter to the buffer threshold.

5. The method according to claim 1, wherein: The server is part of a data service, and the method further comprises transmitting health-related information from the server to a component of the data service.

6. A system for data service overload detection and mitigation, comprising: an overload detection logic component of a server that processes an incoming client request from a client and determines whether subsequent processing of the incoming client request will result in an expected overload condition with respect to request processing capabilities of the server, wherein the expected overload condition is defined by a threshold at which a client request buffer of the server is not full; and If it is determined that subsequent processing of the incoming client request will not result in the expected overload condition, the overload detection logic component adds a request information entry associated with the incoming client request to the client request buffer and increments a counter that tracks pending client requests in the client request buffer, or If it is determined that subsequent processing of the incoming client request will result in the expected overload condition, the overload detection logic component discards the incoming client request and returns a retry response to the client indicating that the client is to retry the corresponding client request after a time delay; and The request information entry associated with the client request is removed from the client request buffer, and for the removed request information entry, the counter is reduced, the request data associated with the removed request information entry is processed to obtain response data, and the response data is returned to respond to the client request corresponding to the removed request information entry.

7. The system according to claim 6, wherein: The server is part of a data service, and wherein the server transmits health-related information to the data service, wherein the health-related information indicates that the server is currently operating within the data service.

8. The system of claim 7, further comprising a load balancer for the data service, the load balancer for the data service coupled to the server, and the incoming client request from the client is received at the server through the load balancer for the data service.

9. The system according to claim 6, wherein: Subsequent processing of the incoming client request is determined to result in the expected overload condition, and wherein the retry response includes time delay data specifying how long the client is to delay before retrying the corresponding client request.

10. The system according to claim 6, wherein: The retry response includes an HTTP: 509 response.

11. A machine-readable storage medium having machine-executable instructions, wherein the machine-executable instructions, when executed, perform operations, the operations comprising: receiving, at the server, an incoming client request for data; evaluating a request buffer condition of a client request buffer to determine whether the client request results in an expected overload condition with respect to request processing capabilities of the server, wherein the expected overload condition is defined by a client request buffer threshold at which the client request buffer is not full, and If evaluating the request buffer status determines that the incoming client request causes the expected overload condition, discarding the incoming client request and returning a retry response to the client, the retry response indicating that the client is to retry the corresponding client request after a time delay; or If evaluating the request buffer status determines that the incoming client request does not result in the expected overload condition, a request information entry associated with the incoming client request is added to the client request buffer and the request buffer status is updated.

12. The machine-readable storage medium according to claim 11, wherein: The time delay is specified in the retry response.

13. The machine-readable storage medium according to claim 11, wherein: Evaluating the request buffer condition includes evaluating an overload counter that tracks the number of entries in the client request buffer and comparing the overload counter to the client request buffer threshold.

14. The machine-readable storage medium according to claim 13, wherein: Updating the request buffer status includes incrementing the overload counter in conjunction with adding the request information entry associated with the incoming client request to the client request buffer.

15. The machine-readable storage medium of claim 14, having other operations, the other operations comprising: Request information entries associated with client requests are removed from the client request buffer, and the overload counter is decremented for the removed request information entries.

16. The machine-readable storage medium of claim 15, having other operations, the other operations comprising: The request data associated with the removed request information item is processed to obtain response data, and the response data is returned in response to a client request corresponding to the removed request information item.

17. The machine-readable storage medium according to claim 11, wherein: Returning the retry response to the client includes returning an HTTP:509 response.

Images