Rule violation detection

Abstract

A method (1200) for detecting traffic rise rule violations includes receiving data element retrieval requests (105) from an information retrieval system (150) and determining a requests per second (RPS) (510) for a key range (430). The method also includes determining a moving average of the RPS (520) for the key range. The method further includes determining a number of incremental violations (312), each incremental violation including a respective start time instance (530) when the RPS exceeds an incremental RPS limit. For each incremental violation, the method includes determining a maximum consistent load (930) and determining whether the RPS exceeds the maximum consistent load based on the start time instance of the respective incremental violation. When the RPS has exceeded the maximum consistent load, the method includes determining that the incremental violation corresponds to a full history violation (332), the full history violation (332) indicating a performance degradation of the information retrieval system.

Description

[0001] This application is a divisional application of the following application:

[0002] International application number of the original application: PCT / US2020 / 061879

[0003] The original international filing date was November 23, 2020.

[0004] The original application's national application number was 202080082053.6.

[0005] Original invention title: Rule violation detection Technical Field

[0006] This disclosure relates to rule violation detection, such as the 500 / 50 / 5 traffic increase rule. Background Technology

[0007] Cloud computing and storage are rapidly gaining popularity. As more users and businesses migrate their data to distributed storage systems, the performance degradation of cloud services is becoming increasingly significant. Large data stores (e.g., databases) are often distributed across multiple servers and horizontally partitioned (i.e., shards) to improve performance. Database sharding typically allows servers to handle dynamic loads by adding and removing resources in near real-time. However, there is a non-zero response time for adding resources, and a sufficiently rapid increase in traffic can overload the underlying storage and lead to performance degradation. Summary of the Invention

[0008] One aspect of the present invention provides a method for detecting traffic increase rule violations. The method includes receiving data element retrieval requests at data processing hardware, wherein each request originates from at least one data element of an information retrieval system. The information retrieval system includes multiple data elements. The method further includes determining, by the data processing hardware, requests per second (RPS) of the key range of the information retrieval system based on the number of received data element retrieval requests. The method further includes determining, by the data processing hardware, a moving average of the RPS of the key range of the information retrieval system over a first time period based on the number of received data element retrieval requests, and determining, by the data processing hardware, the number of delta violations. Each delta violation includes a corresponding instance in time when the RPS exceeds a delta RPS limit. The delta RPS limit is based on the moving average of the RPS. For each delta violation, the method includes: determining, by the data processing hardware, the maximum conforming load of the key range over a second time period, and determining, by the data processing hardware, whether the RPS exceeds the maximum conforming load of the key range based on the start time instance of the corresponding delta violation. The method also includes, when the RPS exceeds the maximum consistency load of the key range, the data processing hardware determines that the incremental violation corresponds to a full-history violation. A full-history violation indicates a performance degradation of the information retrieval system.

[0009] Implementations of this disclosure may include one or more of the following optional features. In some implementations, determining whether the RPS exceeds the maximum consistency load of the key range based on the start time instance of the corresponding incremental violation includes determining a zero-historical RPS limit by the data processing hardware. The zero-historical RPS limit includes a function of the minimum of the historical zero-historical RPS limit and a moving average of the RPS. The method may also include determining by the data processing hardware the duration that begins at the start instance of the corresponding incremental violation and ends when the RPS no longer exceeds the zero-historical RPS limit; the method may optionally include determining a zero-historical violation by the data processing hardware based on the duration for which the RPS exceeds the zero-historical RPS limit. In some examples, when the RPS exceeds the maximum consistency load of the key range, the method includes determining by the data processing hardware that the zero-historical violation corresponds to one or more full-historical violations.

[0010] In some implementations, the method further includes generating a request grid comprising multiple cells by data processing hardware. Each cell represents a time bucket of fixed length, and each key bucket includes a key range whose size is set to represent the average number of requests selected. Determining a moving average of RPS may include determining a moving average for each key bucket. Optionally, the size of each key bucket is set to have the same average number of requests. In some examples, determining the number of incremental violations includes determining narrowband candidate incremental violations, where each narrowband candidate incremental violation represents a time instance when the RPS of the corresponding key bucket exceeds an incremental RPS limit. The method may also include determining wideband candidate incremental violations based on narrowband candidate incremental violations, where each wideband incremental violation includes adjacent narrowband incremental violations, and for each wideband candidate incremental violation, determining whether the wideband candidate incremental violation is an incremental violation based on the number or intensity of requests.

[0011] In some implementations, the maximum consistency load includes the maximum load that the information retrieval system can maintain without degrading performance. In some examples, the information retrieval system includes a dynamically range-fragmented information retrieval system. Determining the maximum consistency load based on the number of data element retrieval requests previously received during a threshold time window may include determining the maximum consistency load. Optionally, between each adjacent pair of incremental violations, the RPS does not exceed an incremental RPS limit.

[0012] Another aspect of this disclosure provides a system for detecting traffic increase rule violations. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving data element retrieval requests, each request requesting at least one data element from an information retrieval system. The information retrieval system includes multiple data elements. The operations also include determining the requests per second (RPS) of the key range of the information retrieval system based on the number of received data element retrieval requests. The operations further include determining a moving average of the RPS of the key range of the information retrieval system over a first time period based on the number of received data element retrieval requests, and determining the number of incremental violations. Each incremental violation includes a corresponding instance-in-time (IIT) when the RPS exceeds an incremental RPS limit. The incremental RPS limit is based on the moving average of the RPS. For each incremental violation, the operations include determining the maximum consistency load of the key range over a second time period, and determining whether the RPS exceeds the maximum consistency load of the key range based on the IIT instance of the corresponding incremental violation. The operations also include determining that when the RPS exceeds the maximum consistency load of the key range, the incremental violation corresponds to a full-history violation. A full-history violation indicates performance degradation of the information retrieval system.

[0013] This aspect may include one or more of the following optional features. In some implementations, determining whether the RPS exceeds the maximum consistency load of the key range based on the start time instance of the corresponding incremental violation includes determining a zero-historical RPS limit. The zero-historical RPS limit includes a function of the minimum of the historical zero-historical RPS limit and a moving average of the RPS. The operation may also include determining the duration that begins at the start instance of the corresponding incremental violation and ends when the RPS no longer exceeds the zero-historical RPS limit; this operation may optionally include determining the zero-historical violation based on the duration for which the RPS exceeds the zero-historical RPS limit. In some examples, when the RPS exceeds the maximum consistency load of the key range, the operation includes determining that the zero-historical violation corresponds to one or more full-historical violations.

[0014] In some implementations, the operation further includes generating a request grid comprising multiple cells. Each cell represents a time bucket of fixed length, and each key bucket includes a key range whose size is set to represent the average amount of requests selected. Determining a moving average of RPS may include determining a moving average for each key bucket. Optionally, the size of each key bucket is set to have the same average amount of requests. In some examples, determining the number of incremental violations includes determining narrowband candidate incremental violations, where each narrowband candidate incremental violation represents a time instance when the RPS of the corresponding key bucket exceeds an incremental RPS limit. The operation may also include determining wideband candidate incremental violations based on the narrowband candidate incremental violations, where each wideband incremental violation includes adjacent narrowband incremental violations, and for each wideband candidate incremental violation, determining whether the wideband candidate incremental violation is an incremental violation based on the number or intensity of requests.

[0015] In some implementations, the maximum consistency load includes the maximum load that the information retrieval system can maintain without performance degradation. In some examples, the information retrieval system includes a dynamically range-fragmented information retrieval system. Determining the maximum consistency load based on the number of data element retrieval requests previously received during a threshold time window may include determining the maximum consistency load. Optionally, between each adjacent pair of incremental violations, the RPS does not exceed an incremental RPS limit.

[0016] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0017] Figure 1A and 1B This is a schematic diagram of an example system that provides rule violation detection for information retrieval systems.

[0018] Figure 2 This is a diagram illustrating the horizontal splitting of data objects.

[0019] Figure 3 This is a schematic diagram of an exemplary component of a rule violation detector.

[0020] Figure 4 It is a schematic diagram of a grid of grid elements in the key space representing at least a portion of a reference window for a request to an information retrieval system.

[0021] Figure 5 It is a schematic diagram of the graphs of requests per second (RPS), RPS load, and incremental RPS limit.

[0022] Figure 6 This is a schematic diagram of another graph showing RPS, RPS load, and incremental RPS limit.

[0023] Figure 7 This is a schematic diagram of another graph showing RPS, RPS load, and incremental RPS limit.

[0024] Figure 8 It is a schematic diagram of the curves of RPS, RPS load and zero-history RPS limit.

[0025] Figure 9 It is a schematic diagram of the curves of RPS, RPS load, and all-history RPS limit.

[0026] Figure 10 This is a diagram illustrating the relationship between the reference window, RPS load, and the full-history RPS limit.

[0027] Figure 11 This is a diagram of the bootstrap window used for bootstrapping the full history RPS limit.

[0028] Figure 12 This is a flowchart illustrating an exemplary setup of a method for detecting violations of traffic increase rules.

[0029] Figure 13 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0030] In the various figures, the same reference numerals denote the same elements. Detailed Implementation

[0031] The implementation described herein aims to detect traffic spikes in a distributed storage system (e.g., cloud storage) that exceed one or more flow path rules or otherwise indicate a possibility of performance degradation. A rule violation detector detects violations of traffic spike rules and receives data element retrieval requests (i.e., storing or retrieving data) for an information retrieval system executed on the distributed storage system. The rule violation detector may receive a portion of the data element retrieval requests (i.e., sample the requests) sufficient to characterize all data element retrieval requests destined for the information retrieval system and determine requests per second (RPS) for the key range of the information retrieval system. The rule violation detector determines a moving average of the RPS for the key range. Using the RPS and the moving average of RPS, the violation detector determines the number of incremental violations over a time period. An incremental violation includes a time instance where the moving average RPS exceeds the RPS limit. For each incremental violation, the violation detector determines the maximum consistency load of the key range and determines whether the moving average of the RPS exceeds the maximum consistency load during the duration of the corresponding incremental violation. When the moving average of the RPS exceeds the maximum consistency load of the key range, the violation detector determines that the incremental violation corresponds to a traffic spike rule violation.

[0032] Distributed storage systems often partition databases to easily allow for scaling by distributing the load across multiple machines. Databases can be partitioned vertically or horizontally. Horizontally partitioned databases are also known as sharded databases. Each individual partition is called a shard, and each shard can be stored on a separate server or other independent computing device. Sharding a database involves splitting the database's rows into multiple distinct tables. This is the opposite of splitting columns into different tables (e.g., vertical partitioning). Each partition can have the same columns but different rows. Many different sharding techniques can be implemented. For example, key-based sharding, range-based sharding, and directory-based sharding are typical sharding implementations.

[0033] Range-based sharding involves sharding a database based on a range of key values ​​associated with the data. That is, each row in the database is associated with a shard key, and the value of the associated shard key determines which shard that row of data is stored on. A continuous range of shard key values ​​can be assigned to each shard. Therefore, data with a "close" shard key value may be stored in the same shard, which can improve the efficiency of some requests, operations, or queries.

[0034] Some distributed storage systems (e.g., range-sharded storage systems) can adjust key-range load balancing based on changes in input load. This can be referred to as automatic key-range capacity scaling. As previously mentioned, the entire key space is divided into multiple shards (i.e., key-range partitions), which are the units of load balancing. A sustained increase in load on a shard over a threshold time period can trigger a "split" operation, where the shard is split into two or more shards (e.g., adding one or more additional servers or expanding a shard on an existing server), which can effectively double (or more) the capacity of the key range. In some implementations, adjacent key-range partitions experiencing low load levels over a period of time are merged together (i.e., a "merge" operation) to reclaim unused resources. In some examples, the time period for a split operation is much shorter than that for a merge operation. For example, the time period for a split operation can be on the order of 5 minutes, while the time period for a merge operation can be on the order of a day. Therefore, the effective capacity of any key range can be doubled rapidly, and the resulting capacity can be maintained for a long time to prevent additional spikes in the data.

[0035] Because there is a delay before capacity can be increased via a split operation (e.g., every 5 minutes), a sufficiently large and rapid increase in user traffic to a single shard (i.e., a key-range split) can lead to performance degradation. This splitting and merging behavior of a key-range capacity autoscaling system can be modeled using traffic increase rules. User traffic can be compared to the model to identify violations of these rules. Violations of traffic increase rules can indicate performance degradation. In some examples, such violations will affect the cloud service provider's guarantees (i.e., performance guarantees) to the client responsible for the violation. Examples of traffic increase rules could be rules like the 500 / 50 / 5 rule. With the 500 / 50 / 5 rule, a client can start operating on any key range of 500 RPS and increase traffic by 50% every 5 minutes from there. Following this rule ensures that the system can add resources quickly enough to prevent performance degradation.

[0036] In some examples, traffic rise rules are modeled on a traffic limit that user traffic must be kept below (e.g., in RPS) to avoid performance degradation of the information retrieval system. Due to the splitting and merging nature of key range partitioning, traffic limits inherently have a historical dependency on previous traffic data and the existing traffic limits. Determining violations of traffic rise rules is often a very computationally expensive effort due to the large amounts of data and the historical dependency on traffic limits.

[0037] Reference Figure 1A and 1BIn some implementations, system 100 includes one or more client devices 120, 120an associated with one or more clients / consumers / owners 10, 10a-n, which can communicate with remote system 140 via network 130. Remote system 140 may be a distributed system (e.g., a cloud environment) with scalable / elastic resources 110. Resources 110 include computing resources 112 and / or storage resources 114. An information retrieval system 150 (e.g., a distributed storage system or data store) overlays the storage resources 114 to allow one or more client devices 120 to scalably utilize the storage resources 114. Information retrieval system 150 is configured to store data objects 200, 200an from client devices 120. In some examples, data objects include databases (i.e., rows and columns of data elements 202, 202an).

[0038] Remote system 140 receives data object 200 from client device 120 and stores data object 200 on storage abstraction 150 by chunking or splitting data object 200 into constituent data elements or data blocks 202 and storing the data elements on one or more shards 152, 152a-n. Alternatively, remote system 140 generates empty data object 200 and receives data elements 202 from client device 120 to store within data object 200. As used herein, each shard 152 represents a separate compute resource 112 and a horizontally partitioned storage resource 114 (e.g., a separate server) storing data object 200. Remote system 140 can implement any number of shards 152 and can dynamically add or remove shards 152 based on usage.

[0039] Remote system 140 receives data element retrieval request 105 from client 10 to retrieve one or more data elements 202 from information retrieval system 150. Information retrieval system 150 determines the location of data element 202 based on the shard key value 220 (i.e., which shard the data element 202 is stored in) provided in each data element retrieval request 105. Once the appropriate shard 152 is determined, information retrieval system 150 retrieves the appropriate data elements 202 and returns them to the requesting client 10. Data element retrieval request 105 may also include a request to add or update an existing data element 202 to information retrieval system 150. Similarly, information retrieval system 150 can use the associated shard key value 220 included in retrieval request 105 to determine the appropriate shard 152 for storing the uploaded data element 202.

[0040] Reference Figure 1BIn some implementations, the distributed system 140 includes loosely coupled memory hosts 110, 110a-n (e.g., computers or servers), each memory host 110, 110a-n having computing resources 112 (e.g., one or more processors or central processing units (CPUs)) that can communicate with storage resources 114 (e.g., memory hardware, flash memory, dynamic random access memory (DRAM), phase-change memory (PCM), and / or disks) that can be used to cache data. A storage abstraction 150 overlaying the storage resources 114 allows one or more client devices 120, 120a-n to scalably utilize the storage resources 114. Client device 120 can communicate with memory host 110 via network 130 (e.g., via remote procedure call (RPC)).

[0041] In some implementations, the distributed system 140 is “one-sided.” “One-sided” means that most request processing on the memory host 110 can be accomplished by hardware rather than by software executing on the CPU 112 of the memory host 110. Other concepts and features associated with a one-sided distributed cache system can be found in U.S. Patent 9,164,702, the entire contents of which are incorporated herein by reference.

[0042] Distributed system 140 can store constituent data elements or data blocks 202 of data object 200 uploaded by client device 120 on storage resources 114 (e.g., memory hardware) of remote memory host 110 (e.g., storage abstraction 150), and obtain data blocks 202 from remote memory host 110 via RPC or via a network interface controller (NIC) 116 supporting Remote Direct Memory Access (RDMA). The network interface controller 116 (also referred to as a network interface card, network adapter, or LAN adapter) can be a computer hardware component that connects computing devices / resources 112 to network 130. Both memory hosts 110a-n and client devices 120 may have a network interface controller 116 for network communication. Information retrieval system 150, executing on the physical processor 112 of hardware resource 110, registers a set of remote direct memory access regions / locations 118, 118a-n of storage resources (e.g., memory) 114 with the network interface controller 116. Each memory location 118 is configured to store the corresponding data element 202.

[0043] Now for reference Figure 2An exemplary data object 200 (i.e., a table) includes multiple data elements 202 delimited by columns 210, including a first name column 210a, a last name column 210b, a title column 210c, and a shard key value column 220. Here, for simplicity, table 200 has three rows, but in practice, the table can have up to three billion rows. As previously described, remote system 140 can use horizontal partitioning to separate data object 200 and store partitions on separate shards 152. In this example, based on shard key values ​​220, data object 200 is partitioned and stored in shards 1150a, 2150b, and 3150c. In this example, shard 1150a stores data elements 202 associated with shard key values ​​220 between 0 and 10. That is, all data elements associated with shard key values ​​220 between 0 and 10 can be stored in shard 1150a. Shard 2 150b stores data element 202 associated with shard key value 220 between 11 and 30. Similarly, shard 3 150c stores data element 202 associated with shard key value 220 between 31 and 40. These values ​​are merely exemplary and it should be understood that any value (with or without any relation to the rest of data object 200) can be assigned to shard key value 220, and each shard 152 can support any range of shard key values ​​220. For example, shard key values ​​220 can be assigned to existing columns (e.g., employee number, project cost, location field, etc.) of data element 202 within data object 200. The relationship between shard key values ​​220 and data elements (e.g., shard key cardinality, shard key frequency, etc.) can affect the performance of information retrieval system 150.

[0044] Still referencing Figure 2 Data object 200 is divided into three sub-data objects 201, 201ac, each sub-data object having multiple data elements 202. Here, the row with shard key value 220 "5" is stored in shard 1150a, while the row with shard key value 220 "13" is stored in shard 2150b. Similarly, the row with shard key value 220 "28" is stored in shard 3150c. In this way, data object 200 is divided into three shards 150a, 150b, and 150c (each shard can be a separate computing resource), thereby increasing the access capability of information retrieval system 150 to data object 200.

[0045] Come back for reference Figure 1AThe remote system 140 executes rule violation detector 300. As discussed in more detail below, rule violation detector 300 monitors client data element retrieval requests 105. Because the number of requests 105 can be very large, in some examples, rule violation detector 300 randomly samples a portion of requests 105 to represent all of them. Rule violation detector 300 uses the sampled requests 105 to determine whether request 105 violates traffic increase rule 302. Traffic increase rule 302 outlines, for example, the maximum rate at which user traffic (i.e., request 105) can increase over any key range to avoid performance degradation.

[0046] Now for reference Figure 3 In some implementations, the rule violation detector 300 includes an incremental detector 310, a zero-history detector 320, and a full-history detector 330. In some examples, the sampling request 105 may not characterize all trends in the traffic, but only the most significant or notable load changes in the traffic. Various sampling techniques can be implemented to help control and / or minimize sampling errors. The rule violation detector 300 receives the sampling request 105 at the incremental detector 310. In some examples, the incremental detector 310 at least with reference window 404 ( Figure 4 The same time period is used to receive sampling requests 105 (also referred to here as "traffic"). In some examples, reference window 404 is equivalent to the amount of time that information retrieval system 150 will maintain capacity after a split operation minus the amount of time required for information retrieval system 150 to perform the split operation. The period during which system 150 maintains capacity is referred to here as RPS history, and the period during which system 150 increases capacity is referred to here as RPS latency 726 ( Figure 7 For example, when information retrieval system 150 needs five (5) minutes to perform a split operation (i.e., increase capacity in response to increased traffic) and holds the increased capacity for 25 hours before performing a merge operation, reference window 404 can have a duration of 24 hours and 55 minutes. As used herein, the terms “requests per second (RPS)” and “queries per second (QPS)” are used interchangeably.

[0047] Because traffic increase rule 302 has historical dependency in the past, similar to RPS history, the size of reference window 404 is determined to include all necessary history from the current time. In other words, due to the historical dependency of traffic increase rule 302, to determine if traffic increase rule 302 violates at time ts, all traffic and traffic limits from the previous time period (25 hours in this example) are used, equal to the RPS history. In some examples, the most recent time period may not be relevant (e.g., RPS delay 726) because the system has not yet had time to respond to the traffic increase. Therefore, reference window 404 provides a sliding window of time to determine violations on time instances. It should be understood that the specific values ​​used herein are exemplary and can be replaced with any values ​​unless otherwise stated. The sampling request 105 received by incremental detector 310 may represent all traffic during reference window 404.

[0048] Now for reference Figure 4 In some implementations, the sampled traffic is placed in a grid 400 of grid elements 402, 402a-n. The x-axis of the grid can represent time in seconds, and each grid element 402 represents a fixed-length “time bucket” 420, 420a-n. Each time bucket 420 represents all sampled traffic occurring within a set amount of time. For example, each time bucket 420 (i.e., the x-axis of each grid element 402) can represent 10 seconds. The length of the time bucket 420 is adjustable. When the reference window 404 is 24 hours and 55 minutes (i.e., RPS history minus RPS delay 726), the grid 400 will have an x-axis length of 8970 elements (one grid element 402 for every 10 seconds in the 24 hours and 55 minutes reference window 404). In some examples, the incremental detector 310 sequentially grids portions of the reference window 404 to minimize the size of the grid 400. That is, the length of the grid 400 may represent only a portion of the reference window 404. In some examples, the length of the grid can be a ratio of the reference window 404. For example, the incremental detector 310 can generate a grid 400 for the first two hours of the reference window 404 and analyze or process the data. After that, the incremental detector 310 can generate a grid 400 for the next two hours of the reference window 404, and so on, until the entire reference window 404 is processed. In some implementations, the generated grid 400 may not be perfectly aligned or matched at the keyway boundaries. By adjusting the grid generation parameters and using approximation techniques, the necessary metrics can be derived on the grid 400 with an acceptable level of error.

[0049] The y-axis of grid 400 can represent at least a portion of the key space 410 of the information retrieval system 150. Key space 410 represents all potential sharded key values ​​220, and the y-axis, in some examples, represents the entire key space 410 (i.e., from the smallest sharded key value 220 to the largest sharded key value 220), and in other examples, only the relevant portion of key space 410. Each grid element 402 can represent key buckets 430, 430an (i.e., the “height” of grid element 402 represents the range of keys). Key bucket 430 is defined as a narrow range or key band with a start key and an end key. For example, if key space 410 consists of all integers between 0 and 999, then a key bucket 430 can represent sharded key values ​​between 0 and 9. In other words, each grid element 402 represents a slice of the total key space 410 (i.e., key bucket 430) over a period of time (i.e., time bucket 420). In some examples, each key bucket 430 has the same size (i.e., representing an equal amount of key space 410). In other examples, the size of each key bucket 430 is set such that each key bucket 430 represents approximately the same amount of traffic. For example, each key bucket 430 may represent a key space that experiences an average of approximately 50 RPS over the length of grid 400 (which may be part of a reference window 404, such as 2 hours). It is not important that each key bucket 430 represents exactly the same amount of traffic and a rough estimate. Requests per second can also be referred to as queries per second (QPS). Therefore, each grid element 402 of grid 400 may represent multiple requests 105 for a given key range (i.e., key bucket 430) over a given time period (i.e., time bucket 420). At that time bucket 420, that number of requests 105 may be referred to as the RPS 510 of key bucket 430. In some examples, RPS 510 is equivalent to the average RPS multiplied by the length of time bucket 420. For example, when the average RPS is 50 and each time bucket represents 10 seconds, each grid cell 402 will represent 500 requests.

[0050] Using grid 400, incremental detector 310 can determine the moving average for each key bucket 430. The moving average is referred to in this paper as RPS load 520. Figure 5 The moving average (i.e., RPS load 520) is calculated over a period of time called RPS load 522. Figure 5 The RPS load is determined within a time period. In some examples, the RPS load 522 lasts for five minutes. Here, for each grid cell 402, the moving average of the previous five minutes is determined as the RPS load 520 of the corresponding key bucket 430. In some implementations, the incremental detector 310 determines the RPS load 520 whenever the RPS 510 ( Figure 5An incremental violation occurs when, at some point in time, the RPS load exceeds 520 and is equal to the RPS delay 726 prior to that point in time multiplied by the incremental weight 524 (e.g., 1.5). Figure 5 The RPS load 520 multiplied by the incremental weight 524 at a certain point in time (i.e., ts-RPS delay) after an RPS delay of 726 (i.e., ts-RPS delay) can be called the incremental RPS limit 530 at time ts. Figure 5 In other words, the RPS limit at timestamp ts can be the RPS load 520 at the incremental weight 524 multiplied by (ts - RPS delay). The incremental weight 524 can simulate a capacity increase after a split operation (i.e., adding one or more shards). In some examples, RPS 510 can be a second moving average over a different time length than RPS load 520. That is, in some examples, the incremental violation 312 can be determined based on two moving averages with different lengths (e.g., 5 minutes and 1 minute).

[0051] Now for reference Figure 5 Curve 500, with x-axis time (in seconds) and y-axis RPS (in RPS), illustrates the relationship between RPS 510, RPS load 520, and incremental RPS limit 530 when user traffic increases at its maximum without violating the incremental RPS limit 530. From time 0 up to approximately 750 seconds, RPS 510 remains constant at "500". Because RPS 510 is constant, RPS load 520 (i.e., the moving average of RPS 510) is also constant at "500". In this example, the increment 524 equals approximately "1.5", therefore, the incremental RPS limit 530 equals "500" multiplied by "1.5" (i.e., 750). At time = 750 seconds, RPS 510 increases to the incremental RPS limit 530 "750". After a period of time (i.e., RPS latency 726), the incremental RPS limit 530 increases, and RPS 510 (i.e., user traffic) and RPS load 520 increase to match the incremental RPS limit 530. The increase in the incremental RPS limit 530 follows the increase in RPS 510 due to the RPS latency. In other words, this lag models the behavior of the information retrieval system 150 because the system 150 will have latency before additional resources can be added (e.g., adding shards 152). Figure 6 Curve 600 is shown with the same data as curve 500, but with a significantly increased time scale for the x-axis. This time scale makes it clear that the incremental RPS limit 530 increases at an approximately exponential rate. Figure 5 and 6In the example shown, RPS 510 will never exceed the incremental RPS limit 530, so the incremental detector 310 will not detect the existence of the incremental violation 312.

[0052] Now for reference Figure 7 Another curve, 700, provides the same... Figure 5 and 6 Examples of different traffic patterns. Here, the x-axis is again time (in seconds), and the y-axis is requests per second. At time 0, RPS 510 and the incremental RPS limit 530 stabilize at “500” and “750”, respectively. In this case, at time ≈ 500 seconds, user traffic (represented by RPS 510) rapidly increases to “6000”, which exceeds the incremental RPS limit 530 of “750”. Then, user traffic returns to 500 RPS at time ≈ 3000 seconds. Shortly after the first traffic spike (i.e., RPS delay 726), the incremental RPS limit 530 increases to “9000” (i.e., 6000 multiplied by an incremental weight of 1.5 524). In this example, the incremental detector 310 will detect incremental violation 312, which begins at the traffic spike at time ≈ 500 seconds and continues until RPS 510 no longer exceeds the incremental RPS limit 530 (at ≈ 1000 seconds). The increase in incremental RPS limit of 530 (due to the increase in RPS load of 520) makes the remainder of the traffic spike (from t≈1000 seconds to t≈3000 seconds) not an incremental violation 312. After the traffic spike, RPS load of 520 returns to 500 RPS, and incremental RPS limit of 530 returns to 750 RPS.

[0053] Return to reference Figure 3 While incremental violation 312 does not necessarily lead to a violation of the traffic surge rule (as explained in more detail below), incremental detector 310 can effectively analyze large amounts of data (i.e., user traffic) and quickly pinpoint traffic cycles to specific key ranges that can indicate a traffic surge rule violation (i.e., full-history violation 332). In other words, incremental detector 310 detects all areas of traffic that may experience a sudden increase in load. Incremental detector 310 can detect incremental violation 312 in almost any range of key space 410.

[0054] Incremental detector 310 can also determine the widest key range violation 312 (i.e., wideband incremental violation) by combining narrower key range violations (i.e., key bucket 430 or narrowband incremental violation). In other words, when a heavy load increase corresponds to a wide key range, a narrower key range, when combined with other incremental violations 312 (in the key space) that are close to each other, constructs a wider key range and experiences a similar trend in traffic. Therefore, incremental detector 310 can identify incremental violations 312 as candidates for violations of traffic increase rule 302. That is, incremental detector 310 can identify narrowband candidate incremental violations 312 and identify wideband candidate incremental violations from the narrowband candidate incremental violations. Incremental detector 310 can apply additional traffic rules to determine whether the combined incremental violations 312 (i.e., wide key range incremental violations) have experienced an influential increase in traffic load. For example, incremental detector 310 considers the intensity of the increased load in addition to the peak value to provide additional filtering.

[0055] Therefore, the incremental detector 310 detects incremental violations 312 across the entire key space 410 and constructs wideband incremental violations based on narrowband incremental violations. The incremental detector 310 avoids the need for restrictive derivations over arbitrary key ranges, which greatly reduces computational complexity and significantly improves the search space for traffic-increasing rule violations.

[0056] Continue to refer to Figure 3 Incremental detector 310 sends any detected incremental violations 312 and any associated data (i.e., timestamps, key ranges, etc.) to zero-history detector 320. In some implementations, incremental violations 312 do not fully capture the dynamics of the key range capacity autoscaling system model because incremental violations 312 only identify areas in grid 400 where flow levels have increased significantly. That is, a violation of flow rise rule 302 begins with incremental violation 312, but not every incremental violation 312 corresponds to a full-history violation 332.

[0057] For simplicity, the incremental detector 310 uses an incremental RPS limit 530 (i.e., effectively reducing the search space); however, in some examples, the incremental RPS limit 530 does not represent the actual rate of increase of the RPS limit for the model of the key range capacity autoscaling system. Instead, a zero-history RPS limit 830 is used. Figure 8This can be used to more accurately simulate traffic rule 302. For example, instead of multiplying the RPS delay time before a certain time instance (i.e., the amount of RPS delay time before the timestamp of interest) by the incremental weight 524 (i.e., the incremental RPS limit 530), the zero-historical RPS limit 830 can instead be a function of the minimum of the RPS load 520 and the zero-historical RPS limit 830 at the same time (i.e., the historical zero-historical RPS limit 830) multiplied by the RPS increase rate 824. Figure 8 In other words, the increase in the zero-history RPS limit of 830 can be overridden by the previous limit, and therefore may take longer to respond to large traffic spikes.

[0058] Now for reference Figure 8 Curve 800 illustrates the relationship between RPS 510, RPS load 520, and the zero-history RPS limit 830. Curve 800 again has time on the x-axis (in seconds) and RPS on the y-axis. Here, the RPS increase rate 824 is equal to "1.5". Although the RPS increase rate 824 is the same as the increment 524 in the previous example in this example, these values ​​can be different. At time 0, RPS 510 and RPS load 520 stabilize at 500 RPS. The zero-history RPS limit 830 is 750 RPS because RPS load 520 (i.e., 500) multiplied by the RPS increase rate 824 (i.e., 1.5) equals 750 RPS. At approximately time 500 seconds, RPS 510 momentarily increases to 6000 RPS. After a period of time associated with the RPS load duration 522 (e.g., 5 minutes), RPS load 520 also reaches 6000 RPS. However, in this example, the zero-history RPS limit 830 increases in intervals of RPS delay 726. Due to the sharp increase in RPS 510, the minimum between RPS 510 and the previous zero-history RPS limit 830 is the previous zero-history RPS limit 830, until it exceeds 2000 seconds.

[0059] In this example, the incremental detector 310 will detect an incremental violation 312 that begins at 500 seconds (i.e., when traffic suddenly increases) and lasts for 5 minutes (i.e., RPS load lasts 522). Conversely, a zero-history violation 322 begins at the same point as the incremental violation 312 (i.e., approximately 500 seconds) and has a duration equal to the time period until RPS 510 no longer exceeds the zero-history RPS limit 830 (i.e., approximately 2300 seconds). Therefore, in this example, the zero-history violation 322 is longer in duration than the corresponding incremental violation 312. In some examples, a zero-history violation 322 corresponds to multiple incremental violations 312, so the total number of zero-history violations 322 can be less than or equal to the total number of incremental violations 312. For example, as traffic gradually increases (i.e., RPS 510), the head of each step can be marked as an incremental violation 312, but the entire traffic spike can be detected as a single zero-history violation 332.

[0060] Because all zero-history violations 322 include incremental violations 312 at the header, the previous detection of incremental violations 312 (e.g., by incremental detector 310) greatly simplifies the detection of the corresponding zero-history violations 322. The process can also merge multiple incremental violations into a single zero-history violation.

[0061] Refer again Figure 3 The zero-history detector 320 sends the detected zero-history violation 322 along with any relevant information (e.g., timestamp, key range, etc.) to the full-history detector 330. In some implementations, increasing the capacity of the key range in response to a traffic spike (e.g., via adding fragments 152) takes a relatively short period (e.g., 5 minutes), while the information retrieval system 150 can maintain the increased capacity for a significant period (e.g., 1 day), even if the traffic immediately subsides to pre-spike levels. This behavior can be modeled using a full-history RPS limit 930. Figure 9 Using the full-history RPS limit 930, not all zero-history violations 322 will correspond to full-history violations 332, because zero-history violations 322 do not take into account the history of the entire reference window 404. The full-history RPS limit 930 can be referred to as the maximum consistency load for a key range. The maximum consistency load can depend on the implementation of autoscaling resources (i.e., splitting and merging shards). That is, the key range capacity (i.e., the maximum consistency load for each key range) can be determined by load variations, where autoscaling allows for increases in load levels or the number of partitions corresponding to a given key range due to the amount of data held by the key range.

[0062] In some examples, a zero-history violation 322 comprises multiple full-history violations 332. For instance, a zero-history violation 322 with an interval [zh_start, zh_end] (from one or more incremental violations 312) has a flow pattern for a selection key range that typically increases but is wavy, and has two strong maxima within the interval. Between the maxima, the flow decreases and approaches but is not equal to the zero-history limit 830, resulting in a single violation cycle for the zero-history violation 322. When determining a full-history violation 332, each timestamp within the zero-history violation interval (e.g., the duration of the time interval in the time bucket 420) can be compared to the maximum consistent load within the current reference window 404, since each timestamp has a different reference window 404 that moves or slides with the timestamp. Based on the historical level of the load within these reference windows 404, the regions around the two maxima can be derived as two separate full-history violation cycles, while the local minimum between each maxima corresponds to the maximum consistent load.

[0063] The full history detector 330 detects a full history violation 332 corresponding to a violation of the traffic increase rule 302. A full history violation 332 may correspond to a zero history violation 322 where the previous level of consistent workload within the corresponding reference window 404 is missing. More specifically, when examining a zero history violation period [zh_start, zh_end], at any given timestamp ts belonging to that period, a full history limit equal to the RPS increase rate 824 multiplied by the maximum consistent workload of the corresponding reference window 404 can be re-derived. This new limit can be used to determine whether the instance is a full history violation 332 in time. In some examples, only a portion (e.g., the strongest portion of the interval) can be processed as a trial. In some examples, the full history detector 330 only processes the region corresponding to the zero history violation 322 in order to detect a full history violation 332. Because any historical dependency on zero-history violation 322 is reset when traffic returns below the zero-history RPS limit 830, and because of the increased capacity that expands in response to increased traffic, zero-history violation 322 may not take into account some eligible workloads.

[0064] In some implementations, the full history detector 330 also receives a grid 400. The high-resolution grid 400 (e.g., generated by the incremental detector 310) enables the full history detector 330 to derive the boundaries of previous load levels for any key range. Each zero-history violation 322 corresponds to a sudden increase in the flow level. When there is no comparable level of non-violation (i.e., consistent) load in the reference window 404, the full history detector 330 determines that the zero-history violation 322 is a full history violation 332.

[0065] Now for reference Figure 9Curve 900 illustrates the relationship between RPS 510, RPS load 520, and the full-history RPS limit 930. As in the previous example, curve 900 has a time x-axis in seconds and a y-axis in RPS. Here, four traffic spikes are shown that rapidly increase traffic from approximately 500 RPS to approximately 4000 RPS. It is assumed that user traffic (i.e., RPS 510) is stable over a long period before time = 0 (e.g., longer than the reference window period 404). Although all four spikes can be detected as incremental violation 312 and zero-history violation 322, the full-history detector 330 can determine that only the first spike is a full-history violation 332 due to the increased capacity maintained (represented by the full-history RPS limit 930).

[0066] Now for reference Figure 10 The main challenge in detecting violations of traffic rise rule 302 (i.e., full-history violation 332) is the unbounded historical dependency of traffic rise rule 302. That is, whether a violation occurs at a particular time depends on traffic behavior prior to that time. This obviously causes problems in determining the initial full-history RPS limit 930 at the beginning of reference window 404. To address this historical dependency, rule violation detector 300 can implement a bootstrapping technique. That is, the full-history RPS limit 930 can be initialized before rule violation detector 300 samples enough data to satisfy the RPS history (e.g., when rule violation detector 300 has sampled less than 25 hours of data).

[0067] Due to the nature of the full-history RPS limit 930, when a given reference window 404 is not violated, it is determined that the full-history RPS limit 930 only requires the RPS load value 520 (not the history of the full-history RPS limit 930). In other words, a reference window 404 without violation can act as a "history reset" event, making the history before the reference window irrelevant.

[0068] Now for reference Figure 11In some implementations, the rule violation detector 300 selects a sufficiently large boot window 1100 to ensure that there is a high probability of no full history violation 332 at least on the reference window 404 with boot window 1100. The rule violation detector 300 may begin at the start of boot window 1100 and determine the full history RPS limit 930 (starting with only RPS load 520). Due to the high probability of a reset event within boot window 1100, the rule violation detector 300 may trust the full history RPS limit 930 at the end of boot window 1100. Therefore, before the rule violation detector 300 detects the full history violation 332 within reference window 404, the rule violation detector 300 can bootstrap the full history RPS limit 930 by selecting a boot window 1100 preceding reference window 404 in time.

[0069] Figure 12 This is a flowchart illustrating an exemplary arrangement of the operation of method 1200 for detecting traffic surge rule violations within an information retrieval system. Method 1200 begins at operation 1202, where data element retrieval requests 105 are received at data processing hardware 112, each request requesting at least one data element 202 from information retrieval system 150. The information retrieval system includes multiple data elements 202. Method 1200 includes, at operation 1204, the data processing hardware 112 determining, based on the number of received data element retrieval requests 105, the requests per second (RPS) of the key range 430 of information retrieval system 150.

[0070] 510. In operation 1206, method 1200 includes determining a key range 430 of the information retrieval system 150 by data processing hardware 112 based on the number of received data element retrieval requests 105 in a first time period (e.g., Figure 5-7 The moving average 520 of RPS (continuous RPS load) on 522. In operation 1208, method 1200 includes determining multiple incremental violations 312 by data processing hardware 112. Each incremental violation 312 includes a corresponding start time instance when RPS 510 exceeds the incremental RPS limit 530. The incremental RPS limit 530 is based on the moving average 520 of RPS.

[0071] For each incremental violation 312, method 1200 includes, in operation 1210, determining by data processing hardware 112 the maximum consistency load 930 of key range 430 over a second time period 404, and in operation 1212, determining by data processing hardware 112 whether RPS 510 exceeds the maximum consistency load 930 of key range 430 based on the start time instance of the corresponding incremental violation 312. When RPS 510 exceeds the maximum consistency load 930 of key range 430, method 1200 includes, in operation 1214, determining by data processing hardware 112 that incremental violation 312 corresponds to full history violation 332. Full history violation 332 represents performance degradation of information retrieval system 150.

[0072] Figure 13 This is a schematic diagram of an example computing device 1300 that can be used to implement the systems and methods described in this document. The computing device 1300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0073] The computing device 1300 includes: a processor 1310 (e.g., data processing hardware), a memory 1320, a storage device 1330, a high-speed interface / controller 1340 connected to the memory 1320 and a high-speed expansion port 1350, and a low-speed interface / controller 1360 connected to a low-speed bus 1370 and the storage device 1330. Each component 1310, 1320, 1330, 1340, 1350, and 1360 is interconnected using different buses, and each component can be mounted on a common motherboard or otherwise installed as needed. The processor 1310 can process instructions that execute within the computing device 1300, including instructions stored in the memory 1320 or the storage device 1330 to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 1380 coupled to the high-speed interface 1340. In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and various types of memory, if needed. Similarly, multiple computing devices 1300 can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0074] Memory 1320 stores information non-temporarily within computing device 1300. Memory 1320 may be a computer-readable medium, volatile memory cells(s), or non-volatile memory cells(s). Non-volatile memory 1320 may be a physical device for storing programs (e.g., instruction sequences) or data (program state information) used by computing device 1300 on a temporary or permanent basis. Examples of non-volatile memory include, but are not limited to: flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as bootloaders). Examples of volatile memory include, but are not limited to: random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and optical discs or magnetic tapes.

[0075] Storage device 1330 provides massive storage for computing device 1300. In some embodiments, storage device 1330 is a computer-readable medium. In various embodiments, storage device 1330 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied as an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1320, storage device 1330, or memory on processor 1310.

[0076] High-speed controller 1340 manages bandwidth-intensive operations of computing device 1300, while low-speed controller 1360 manages lower bandwidth-intensive operations. This functional allocation is merely exemplary. In some embodiments, high-speed controller 1340 is coupled to memory 1320, display 1380 (e.g., via a graphics processor or accelerator) is coupled to high-speed expansion port 1350, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 1360 is coupled to storage device 1330 and low-speed expansion port 1390. Low-speed expansion port 1390 may include various communication ports (e.g., USB, Bluetooth, Ethernet, and wireless Ethernet) and may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers, for example, via a network adapter.

[0077] As shown in the figure, the computing device 1300 can be implemented in various forms. For example, the computing device 1300 can be implemented as a standard server 1300a, or multiple times in a group of such servers 1300a, or as a laptop computer 1300b, or as part of a rack server system 1300c.

[0078] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems and / or optical circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "program," or simply a "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0080] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0081] The processes and logic flows described herein can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. Processes and logic flows can also be executed by special-purpose logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, general-purpose microprocessors, special-purpose microprocessors, and any type of digital computer processor. Generally, the processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are: a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, or the computer may be operatively coupled to receive data from or transfer data to or both of these mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0082] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touchscreen, and optionally including a keyboard and pointing device (e.g., a mouse or trackball), through which the user provides input to the computer. Other types of devices can be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound input, voice input, or tactile input. Additionally, the computer can interact by sending documents to and receiving documents from the device used by the user, for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0083] Several embodiments have been described. However, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method, characterized in that, The method is executed by data processing hardware, causing the data processing hardware to perform operations, including: The system receives multiple data element retrieval requests, each of which requests at least one data element from an information retrieval system, which includes multiple data elements. A request grid comprising multiple units is generated for the information retrieval system. Each unit represents a corresponding time bucket of a corresponding key bucket, and each corresponding key bucket contains a corresponding key range of the information retrieval system. For each corresponding key bucket of the request grid: Based on the received multiple data element retrieval requests, determine the corresponding requests per second (RPS) for the corresponding key range of the corresponding key bucket; Determine the corresponding RPS load, which includes a moving average of multiple data element retrieval requests corresponding to the corresponding key range of the corresponding key bucket over a period of time; and Determine the corresponding incremental RPS limit based on the corresponding RPS load; and For a key bucket: Determine that the corresponding RPS of the key bucket exceeds the corresponding incremental RPS limit of the key bucket; and Based on the determination that the corresponding RPS of a key bucket exceeds the corresponding incremental RPS limit of the key bucket, an incremental violation indicating a potential degradation in the performance of the information retrieval system is identified.

2. The method according to claim 1, characterized in that, The incremental violation includes a start time instance corresponding to the corresponding RPS exceeding the corresponding incremental RPS limit.

3. The method according to claim 2, characterized in that, The operation further includes, for the incremental violation: Determine the maximum consistency load of the key bucket during the second duration; Based on the start time instance of the incremental violation, it is determined that the corresponding RPS exceeds the maximum consistency load of the key bucket; and In response to determining that the corresponding RPS exceeds the maximum consistency load of the key bucket, the incremental violation is determined to correspond to a full history violation, which indicates a performance degradation of the information retrieval system.

4. The method according to claim 3, characterized in that, Based on the start time instance of the incremental violation, determine whether the corresponding RPS exceeds the maximum consistency load of the key bucket, including: Determine a zero-historical RPS limit, which includes the minimum value of the historical zero-historical RPS limit and a function of the corresponding RPS load; A third duration is determined, which begins at the start instance of the incremental violation and ends when the corresponding RPS no longer exceeds the zero-history RPS limit; A zero-history violation is determined based on the duration for which the corresponding RPS exceeds the zero-history RPS limit; and In response to determining that the corresponding RPS exceeds the maximum consistency load of the key bucket, it is determined that the zero history violation corresponds to a full history violation.

5. The method according to claim 3, characterized in that, Determining the maximum consistency load includes: determining the maximum consistency load based on the number of data element retrieval requests pre-received within a threshold time window.

6. The method according to claim 3, characterized in that, The operation also includes: Select the time window preceding the second duration; and Determine the maximum consistency load of a key bucket within the time window.

7. The method according to claim 1, characterized in that, The size of each key bucket is determined to represent the average number of data element retrieval requests for the selected data element.

8. The method according to claim 1, characterized in that, The information retrieval system includes a dynamic range segmented information retrieval system.

9. The method according to claim 1, characterized in that, Determining the corresponding incremental RPS limit for each corresponding key bucket of the requested grid includes determining the corresponding incremental RPS limit based on the corresponding RPS load and the incremental weight indicating the simulated capacity increase after the split operation.

10. The method according to claim 9, characterized in that, The splitting operation includes adding memory that can be used for the corresponding key bucket.

11. A system for detecting violations of traffic increase rules, characterized in that, include: Data processing hardware; and Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including: The system receives multiple data element retrieval requests, each of which requests at least one data element from an information retrieval system, which includes multiple data elements. A request grid comprising multiple units is generated for the information retrieval system. Each unit represents a corresponding time bucket of a corresponding key bucket, and each corresponding key bucket contains a corresponding key range of the information retrieval system. For each corresponding key bucket of the request grid: Based on the received multiple data element retrieval requests, determine the corresponding requests per second (RPS) for the corresponding key range of the corresponding key bucket; Determine the corresponding RPS load, which includes a moving average of multiple data element retrieval requests corresponding to the corresponding key range of the corresponding key bucket over a period of time; and Determine the corresponding incremental RPS limit based on the corresponding RPS load; and For a key bucket: Determine that the corresponding RPS of the key bucket exceeds the corresponding incremental RPS limit of the key bucket; and Based on the determination that the corresponding RPS of a key bucket exceeds the corresponding incremental RPS limit of the key bucket, an incremental violation indicating a potential degradation in the performance of the information retrieval system is identified.

12. The system according to claim 11, characterized in that, The incremental violation includes a start time instance corresponding to the corresponding RPS exceeding the corresponding incremental RPS limit.

13. The system according to claim 12, characterized in that, The operation further includes, for the incremental violation: Determine the maximum consistency load of the key bucket during the second duration; Based on the start time instance of the incremental violation, it is determined that the corresponding RPS exceeds the maximum consistency load of the key bucket; and In response to determining that the corresponding RPS exceeds the maximum consistency load of the key bucket, the incremental violation is determined to correspond to a full history violation, which indicates a performance degradation of the information retrieval system.

14. The system according to claim 13, characterized in that, Based on the start time instance of the incremental violation, determine whether the corresponding RPS exceeds the maximum consistency load of the key bucket, including: Determine a zero-historical RPS limit, which includes the minimum value of the historical zero-historical RPS limit and a function of the corresponding RPS load; A third duration is determined, which begins at the start instance of the incremental violation and ends when the corresponding RPS no longer exceeds the zero-history RPS limit; A zero-history violation is determined based on the duration for which the corresponding RPS exceeds the zero-history RPS limit; and In response to determining that the corresponding RPS exceeds the maximum consistency load of the key bucket, it is determined that the zero history violation corresponds to a full history violation.

15. The system according to claim 13, characterized in that, Determining the maximum consistency load includes: determining the maximum consistency load based on the number of data element retrieval requests pre-received within a threshold time window.

16. The system according to claim 13, characterized in that, The operation also includes: Select the time window preceding the second duration; and Determine the maximum consistency load of a key bucket within the time window.

17. The system according to claim 11, characterized in that, The size of each key bucket is determined to represent the average number of data element retrieval requests for the selected data element.

18. The system according to claim 11, characterized in that, The information retrieval system includes a dynamic range segmented information retrieval system.

19. The system according to claim 11, characterized in that, Determining the corresponding incremental RPS limit for each corresponding key bucket of the requested grid includes determining the corresponding incremental RPS limit based on the corresponding RPS load and the incremental weight indicating the simulated capacity increase after the split operation.

20. The system according to claim 19, characterized in that, The splitting operation includes adding memory that can be used for the corresponding key bucket.

Images