Dynamic slice allocation in near real-time search platform

Abstract

Dynamic slice allocation in near real-time search platforms is described. A computing system may include a cluster including a plurality of nodes, each of the plurality of nodes including a plurality of slices storing searchable data. The query load distribution of the cluster is evaluated in real-time. A first slice of the first node and a second slice of the second node are identified based on the query load distribution. A first slice of a first node is commanded to be exchanged with a second slice of a second node.

Description

Background Art

[0001] The distributed data storage and retrieval system includes a near real-time (NRT) search platform that enables storage, search, and analysis of large amounts of data substantially as the data is generated. By way of example, the NRT search platform can employ a distributed architecture in which data is divided into smaller units called shards that are distributed across multiple nodes within a cluster to enhance performance, scalability, and fault tolerance. Shard balancing can be used to achieve a relatively uniform distribution of data and query loads across nodes in a cluster in order to maintain efficient performance and resource utilization in the cluster. Effective shard balancing enhances search performance and response time, reduces or prevents node overload, and ensures efficient resource utilization. Summary of the invention

[0002] Dynamic shard allocation in a near real-time search platform is described. A computing system may include a cluster, the cluster including a plurality of nodes, each of the plurality of nodes including a plurality of shards storing searchable data. A query load distribution of the cluster is evaluated in real time. A first shard of a first node and a second shard of a second node are identified based on the query load distribution. A first shard of the first node is commanded to be exchanged with a second shard of the second node.

[0003] The present invention summary introduces a series of concepts in a simplified form, which will be further described in the following detailed description. Therefore, the present invention summary is not intended to illustrate the essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is described with reference to the accompanying drawings. The entities represented in the drawings refer to one or more entities, and thus singular or plural forms of the entities may be referred to interchangeably in the discussion.

[0005] Figure 1 is an illustration of a digital media environment in an example implementation that is operable to employ the dynamic slice allocation techniques described herein.

[0006] Figure 2 Depicted are example implementations that illustrate operation of an analysis and management platform to perform dynamic shard allocation in a cluster of a near real-time search platform.

[0007] Figure 3A and Figure 3B An illustrative example of dynamic shard allocation for cluster balancing in a near real-time search platform is shown.

[0008] Figure 4An example of transaction time and processing unit usage of two nodes is shown before and after cluster balancing via dynamic shard allocation has been performed.

[0009] Figure 5 is a flow chart that depicts the algorithm as a step-by-step process in an example of implementing dynamic shard allocation in a near real-time search platform.

[0010] Figure 6 An example system is shown that includes various components of an example device that can be implemented as described with reference to Figures 1 to 5 Any type of computing device described and / or used to implement embodiments of the techniques described herein. DETAILED DESCRIPTION

[0011] Overview

[0012] The near real-time search platform can adopt a distributed architecture that includes a collection (referred to herein as a cluster) of multiple interconnected nodes (e.g., servers) that work together to store and manage data. The nodes are divided into smaller units called shards, which store at least a portion of the index. Shards enable the near real-time search platform to perform search and index operations in parallel. For example, the near real-time search platform can distribute queries across all relevant shards, and each shard processes its portion of the data independently. This parallel processing reduces query response time.

[0013] A given shard can have one or more replica shards (also referred to herein as "replicas"). A replica is an exact copy of the original shard on a separate node, which increases the fault tolerance of the cluster. For example, if a node or shard goes offline or becomes degraded, the replica serves as a backup. In addition, even when the original shard remains online and functional, its replicas can respond to queries through a feature called adaptive replica selection to more evenly distribute the query load throughout the cluster. For example, adaptive replica selection can be employed to improve query performance and reduce response time by selecting the most appropriate replica shard for each query (e.g., a read request) based on the node's current query load, network conditions, etc.

[0014] During creation, new shards are allocated based on the number of shards per node, disk usage, and a watermark configuration to ensure a relatively even distribution of shards across nodes, where the watermark configuration defines lower and upper bounds on disk usage on a given node. However, this allocation strategy does not take into account query load. For example, a first node with higher disk usage may primarily store older, less frequently queried indexes in its shards compared to a second node with lower disk usage but with newer, more frequently queried indexes. Therefore, a new shard may be allocated to a second node even if the second node receives a higher query load than the first node. Although adaptive replica selection takes query load into account, it may not achieve efficient and effective query load balancing depending on the location of the replicas in the cluster.

[0015] Existing shard management techniques primarily rely on predefined criteria for shard creation, reallocation, combination, and deletion. For example, a technique known as index lifecycle management performs actions based on index age and relies on predefined tiers such as "hot," "warm," and "cold" based on age. These techniques typically lack the ability to dynamically adapt to evolving cluster conditions, leading to suboptimal performance and resource imbalance for near real-time search platforms. Additionally, existing approaches are typically designed for append-only use cases and are therefore not broadly applicable to use cases that include modifying or deleting data in an index.

[0016] Thus, dynamic shard allocation in a near real-time search platform is described. The technology enables identification and execution of shard moves that balance the query load of a cluster based on a real-time evaluation of the query load distribution in the cluster, in a manner that is independent of cluster-level node configuration or static cluster ranking criteria. Typically, cluster balance is an NP-hard (non-deterministic polynomial-time hard) optimization problem that cannot be solved using a global or local optimal search algorithm because each move affects the cluster balance in a way that cannot be mathematically predicted. Therefore, according to the technology described herein, dynamic shard allocation includes: planning and executing one or more shard exchanges between high query load nodes and low query load nodes in response to detecting that the query load distribution is unbalanced in the cluster; re-evaluating the query load distribution in the cluster; and executing one or more additional shard exchanges in response to the query load distribution remaining unbalanced.

[0017] In one example, the shard allocation manager evaluates cluster balance or lack thereof by determining the standard deviation of the number of queries per node executed during a defined time interval. A cluster imbalance is detected in response to the highest number of queries per node received during the time interval being greater than the median number of queries per node by at least the standard deviation. In response to detecting a cluster imbalance, the shard allocation manager sorts the nodes and shards based on the query rate per node and per shard during the defined time interval. For example, a high query load node is selected based on the high query load node receiving more queries relative to other nodes of the cluster during the defined time interval, and a low query load node is selected based on the low query load node receiving fewer queries relative to other nodes of the cluster during the defined time interval. For example, a high query load shard is selected from a high query load node based on the high query load shard receiving more queries than other shards of the high query load node. Similarly, for example, a low query load shard is selected from a low query load node based on the low query load shard receiving fewer queries than other shards of the low query load node.

[0018] If indicated, the high query load shard is swapped with the low query load shard, and the cluster balance is reevaluated before planning more shard moves. That is, the high query load shard is written to the low query load node and deleted from the high query load node, and the low query load shard is written to the high query load node and deleted from the low query load node. Swapping a pair of shards between two nodes avoids triggering the near real-time search platform's automatic action to balance the number of shards between nodes. For example, if a high query load shard is moved from a high query load node to a low query load node without also moving the low query load shard, random shards may be moved to the high query load node, which may result in an undesirable impact on the query load distribution.

[0019] Therefore, the technology described in this article overcomes the challenges of conventional technology by enabling real-time analysis of query load to identify the following shard moves: the shard move is expected to bring balance to the query load distribution of the cluster. It is worth noting that the technology described in this article does not rely on cluster-level node configuration and static standards for classifying specific nodes as "hot", "cold", etc., which can change with changing query business patterns and as new indexes are created. Instead, nodes and shards are classified as hot and cold relative to each other based on real-time query load distribution evaluation. In addition, because each shard move affects cluster balance in a way that cannot be mathematically predicted, such as due to adaptive replica selection, cluster balance can be effectively achieved in a way that is suitable for a variety of use case scenarios by performing a small number of shard moves at a time and re-evaluating the query load distribution in the cluster. By balancing the query load distribution in the cluster by moving previously created shards between nodes, query latency of near real-time search platforms can be reduced while more efficiently utilizing the computing resources of the cluster. Further discussion of these and other examples is included in the following sections and is shown in the corresponding figures.

[0020] In the following discussion, an example environment is first described in which the techniques described herein may be employed. Examples of implementation details and procedures that may be performed in the example environment and other environments are then described. Execution of the example procedures is not limited to the example environment, and the example environment is not limited to execution of the example procedures.

[0021] Environment Examples

[0022] Figure 1 1 is an illustration of a digital media environment 100 in an example implementation operable to employ the dynamic shard allocation techniques described herein. The illustrated environment 100 includes a service provider system 102, a computing device 104, and a plurality of client devices 106 communicatively coupled to one another via a network 108. The computing devices implementing the service provider system 102, the computing device 104, and the client devices 106 may be configured in a variety of ways.

[0023] For example, a computing device may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), etc. Thus, computing devices range from full-resource devices with large amounts of memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and / or processing resources (e.g., mobile devices). Additionally, computing devices also represent multiple different devices, such as multiple servers utilized by an enterprise to perform operations "through the cloud," as shown for service provider system 102 and as described with respect to FIG. Figure 6 as described.

[0024] The service provider system 102 includes an executable service platform 110. The executable service platform 110 is configured to enable and manage access to digital services 112 "in the cloud" that are accessible by client devices 106 via a network 108. Thus, the executable service platform 110 provides the underlying infrastructure to manage the execution of the digital services 112, for example by controlling the underlying computing resources.

[0025] The executable service platform 110 supports a number of computing and technical advantages, including the ability of the service provider system 102 to easily scale resources to meet the needs of the entity associated with the client device 106. Thus, instead of incurring the expense of purchasing and maintaining dedicated computer equipment for performing specific computing tasks, cloud computing provides the client device 106 with access to a wide range of hardware and software resources as long as the client has access to the network 108.

[0026] The digital service 112 can take various forms. Examples of digital services include social media services, document management services, storage services, media streaming services, content creation services, productivity services, digital market services, auction services, etc. By way of example, the execution of the digital service 112 by the executable service platform 110 generates data 114, which is shown as being stored in a near real-time search platform 116 (abbreviated as "NRT search platform 116" in the accompanying drawings). The data 114 describes the execution of the digital service 112 in this example and the value of the event that occurred during the execution. The data 114 can be generated, for example, in response to the interaction of the user with the service provider system 102, and can describe the content or list created by the user, the transaction performed by the user, and / or the behavior of the user. In at least one implementation, the data 114 is time series data. The data 114 can also be retrieved from the near real-time search platform 116, for example, in response to the query 120 for use by one or more domain services 118. Non-limiting examples of the one or more domain services 118 include a user experience domain 122 , a user behavior domain 124 , and a search and recommendation domain 126 .

[0027] For example, in the search and recommendation example, data 114 is used by search and recommendation domain 126 to generate recommendations for digital content items. Recommendations can be configured in a manner similar to the following: performing a search based on a search query to locate digital content items. Using search results as recommendations supports a variety of usage scenarios, including digital audio recommendations, digital video recommendations, auction items, etc.

[0028] The near real-time search platform 116 is a distributed data storage and retrieval system that is configured to enable efficient storage and retrieval of data 114. In at least one implementation, the near real-time search platform 116 includes functionality for storing data 114 and enabling data 114 to be searched in near real-time. As used herein, "near real-time" (e.g., NRT) refers to a small delay, such as in the range of 0.5 seconds to 1.5 seconds. Thus, data 114 becomes searchable via the near real-time search platform 116 substantially at the time of creation and without intentional delay.

[0029] In order to store the data 114 in a searchable format, the near real-time search platform 116 includes a cluster 128. The cluster 128 is a collection of one or more nodes (e.g., servers) that stores all of the data 114 and provides joint indexing and search capabilities across all nodes. A node is a single server within the cluster 128 that stores a portion of the data 114 stored in the cluster. In the depicted example, the cluster 128 includes a first node 130 (e.g., "node 1"), a second node 132 (e.g., "node 2"), and a third node 134 (e.g., "node 3"), where the ellipsis indicates that additional nodes may be present in the cluster 128. The nodes are further divided into a plurality of shards that include at least a portion of a separate index. For example, an index may be stored in a single shard, or subdivided into multiple shards, such as when the amount of data in the index exceeds the storage capacity of a single node. An index is a collection of documents having shared characteristics, such as customer data or product data. As a non-limiting example, the data 114 is distributed across daily indexes. For example, a first index includes data generated and stored on a first date, a second index includes data generated and stored on a second date after the first date, and so on.

[0030] exist Figure 1 In the illustrated digital media environment 100, the first node 130 includes a shard 136, the second node 132 includes a shard 138, and the third node 134 includes a shard 140. The first portion of the shards 136, 138, and 140 are primary (e.g., original) shards that handle write operations for their indexes, while the second remaining portions are replicas of the primary shards. Replicas are exact copies of primary shards and are distributed from the corresponding primary shards to different nodes and hardware racks to provide redundancy and fault tolerance. For example, replicas can also respond to read requests, thereby distributing search and retrieval loads across multiple nodes, and provide backup in the event that a primary shard degrades or becomes unavailable. Reference herein, for example, Figure 3A and Figure 3B Describes additional details about the organization of shards in the corresponding nodes.

[0031] The near real-time search platform 116 also includes a cluster controller 142 and a cluster coordinator 144. The cluster controller 142 includes functionality for performing cluster-wide management tasks, such as creating or deleting indexes, managing shard allocations, and maintaining a cluster state 146 for the cluster 128, as described in detail below. The cluster coordinator 144 includes functionality for receiving queries 120 and coordinating the execution of corresponding searches across relevant nodes and shards of the cluster 128. In at least one implementation, the cluster controller 142 and the cluster coordinator 144 are nodes included in the cluster 128. For example, the cluster controller 142 is a master node of the cluster 128.

[0032] The cluster coordinator 144 is configured to receive a query 120 that includes a request to retrieve data stored in the cluster 128 as a query result 148. By way of example, the query 120 may include query terms, criteria, filters, aggregations, and / or other parameters that define what is to be returned as the query result 148. The query 120 may be submitted by a user and / or an application, such as one of the one or more domain services 118. By way of example, the query 120 is submitted through a user-initiated process, via an automatic schedule (e.g., at a predetermined time interval or time of day), or via another type of event trigger.

[0033] In response to receiving the query 120, the cluster coordinator 144 parses the query 120 to determine the search intent, and further determines which shard(s) to query based on the search intent. The cluster coordinator 144 sends the query 120 or a portion thereof to the corresponding node storing the relevant shard (which executes the query 120). In the case of querying more than one shard, parallel processing can be used to execute the query 120 concurrently across multiple shards. The cluster coordinator 144 receives the requested data from the relevant shards and outputs them as query results 148 to the requesting entity.

[0034] The executable service platform 110 also includes an analysis and management platform 150 in electronic communication with the near real-time search platform 116. For example, the analysis and management platform 150 receives data from the near real-time search platform 116, such as an indication of the cluster state 146 and real-time measurements of query load in the cluster 128. The cluster state 146 is a data structure object containing information related to the configuration of the cluster 128, including a list of nodes, indexes, distribution of shards across nodes, and other metadata. The cluster state 146 is managed by the cluster controller 142 and is updated as changes occur in the cluster 128, which are then transmitted to the analysis and management platform 150. The analysis and management platform 150 can store the cluster state 146 as part of the cluster operational data 152 in the storage device 154.

[0035] According to the techniques described herein, the analysis and management platform 150 includes a shard allocation manager 156 that has the following functions: evaluates the real-time query load of the cluster 128 and identifies whether to perform cluster balancing by swapping shards between nodes. By way of example, the shard allocation manager 156 performs cluster balancing based on one or more algorithms stored in non-transitory memory and executed by one or more processors (in Figure 1 The shard allocation manager 156 performs cluster balancing according to the steps of the dynamic shard allocation algorithm 160 shown in Figure 1. Therefore, the shard allocation manager 156 is configured as a dedicated machine for performing dynamic shard allocation. As will be described in detail herein, as part of the dynamic shard allocation algorithm 160, the shard allocation manager 156 performs real-time measurement of the query load in the cluster 128 to determine whether to indicate cluster balance, for example, in response to a query load imbalance greater than a threshold imbalance. In response to indicating cluster balance, the shard allocation manager 156 identifies which shards to exchange so as to more evenly distribute the query load across the nodes of the cluster 128. When identifying the shards, the shard allocation manager 156 sends a command 158 to the cluster controller 142 to instruct the cluster controller 142 to perform a shard exchange. The shard exchange is confirmed by an update to the cluster state 146 received from the cluster coordinator 144.

[0036] Despite Figure 1 150, but the cluster operation data 152 may also include real-time measurements of query load, logs of shards that have been moved during execution of the dynamic shard allocation algorithm 160, and / or another indication that the dynamic shard allocation algorithm 160 has been executed even when shards have not been swapped. Although the storage device 154 is shown as being included in the analysis and management platform 150, it should be understood that the storage device 154 can be remote from the shard allocation manager 156 (e.g., located in a server remote from the shard allocation manager 156) and can be accessed by the shard allocation manager 156 via the network 108.

[0037] In at least one implementation, the analysis and management platform 150 is configured to generate a platform management user interface 162, which is shown as being accessed by the computing device 104 via the network 108 using a communication module 164 (e.g., a browser, a network-enabled application, etc.). The platform management user interface 162 displayed by the display device 166 can also be used to receive input specifying, for example, the state of the dynamic shard allocation algorithm 160 (e.g., enabled or disabled), the execution time and / or timing (e.g., every day at a specified time), and / or the threshold imbalance. In one or more implementations, the analysis and management platform 150 can also output a monitoring notification 168 that can be accessed via the platform management user interface 162. The monitoring notification 168 can indicate that the dynamic shard allocation algorithm 160 was executed and the results of the algorithm, such as whether a shard was swapped, how many shards were swapped, etc. Thus, a user (e.g., a developer) can program and / or adjust the dynamic shard allocation algorithm 160.

[0038] By performing dynamic shard allocation via the analysis and management platform 150, the near real-time search platform 116 is able to execute queries 120 and return query results 148 with reduced latency and without overloading individual nodes. Thus, the performance of the near real-time search platform 116 can be increased, with fewer query timeouts and increased reliability.

[0039] Typically, the functions, features, and concepts described with respect to the examples above and below are adopted in the context of the example processes described in this section. In addition, the functions, features, and concepts described with respect to the different figures and examples in this document are interchangeable with each other and are not limited to the implementation in the context of a specific figure or process. In addition, the boxes associated with the different representative processes and corresponding figures in this document can be applied together and / or combined in different ways. Therefore, the various functions, features, and concepts described with respect to the different example environments, devices, components, figures, and processes in this document can be used in any suitable combination and are not limited to the specific combinations represented by the examples listed in this specification.

[0040] Dynamic Shard Allocation in a Near Real-Time Search Platform

[0041] Figure 2 A more detailed description is given showing Figure 1 An example implementation 200 of operations for the analysis and management platform 150 of FIG. 1 performs dynamic shard allocation in the cluster 128. The following discussion describes techniques that may be implemented utilizing the previously described systems and devices.

[0042] The shard allocation manager 156 of the analysis and management platform 150 executes the dynamic shard allocation algorithm 160, for example, according to a programmed schedule and / or in response to a user-initiated request. The dynamic shard allocation algorithm 160 includes query load evaluation instructions 202, which are executed to perform real-time measurements of the query load distribution of the cluster 128 within a time interval (e.g., duration) defined by the query load evaluation instructions 202. For example, the time interval can be a duration in the range of one minute to three minutes, such as two minutes. In variations, the time interval is less than one minute or greater than three minutes.

[0043] In at least one implementation, the query load evaluation instructions 202 cause the shard allocation manager 156 to send a query load request 204 to the cluster 128, and in response, the cluster 128 sends a query load measurement 206 to the shard allocation manager 156. The query load measurement 206 includes the number of queries received during the time interval by the cluster 128 on a per-node and per-shard basis. For example, the query load measurement 206 specifies the number of queries that each node and each shard within each node has received during the time interval.

[0044] The query load evaluation instructions 202 further cause the shard allocation manager 156 to identify whether a shard move for cluster balancing is indicated based on the query load measurement 206. By way of example, a shard move is indicated in response to the query load distribution of the cluster 128 having at least a threshold imbalance as determined based on the query load measurement 206. For example, the shard allocation manager 156 may calculate a median number of queries per node and a standard deviation of the number of queries per node based on the query load measurement 206, and the shard allocation manager 156 may determine that a threshold imbalance exists in response to the highest number of queries per node being at least a standard deviation greater than the median.

[0045] In response to the cluster 128 having at least a threshold imbalance, the shard allocation manager 156 also evaluates the query load measurement 206 via the node identification instructions 208. The node identification instructions 208 are executed to sort the nodes according to the number of queries, such as from the highest number of queries (and therefore the highest query load) in descending order to the lowest number of queries. Alternatively, the nodes can be sorted in ascending order from the lowest number of queries to the highest number of queries. Therefore, the node identification instructions 208 enable the shard allocation manager 156 to identify the hot nodes that experience the highest query load of the cluster 128 during the time interval and the cold nodes that experience the lowest query load of the cluster 128. With respect to nodes and shards, the terms "hot" and "cold" used herein refer to relative query loads (e.g., number of queries) and are not meant to represent temperature. That is, "hot" refers to a high number of queries relative to other nodes / shards of the cluster 128, while "cold" refers to a low number of queries relative to other nodes / shards of the cluster 128.

[0046] The shard allocation manager 156 also executes shard identification instructions 210 to identify one or more shards of the hot node to exchange with one or more shards of the cold node. The shard identification instructions 210 are executed to sort the identified shards within the hot and cold nodes according to the number of queries (such as in ascending or descending order) to identify the shards that experience the highest query load in the hot nodes and the shards that experience the lowest query load in the cold nodes during the time interval.

[0047] In at least one implementation of the near real-time search platform 116, moving a shard from a hot node will trigger another shard to move to the hot node to keep the number of shards per node balanced. Therefore, when executing the shard identification instruction 210, the shard allocation manager 156 identifies a pair including a high query load shard from a hot node and a low query load shard from a cold node to avoid such automatic and uncontrolled triggering. It should be understood that more than one pair can be exchanged during a given move, and the number of pairs to be exchanged during the move can be a pre-programmed value. For example, the number of pairs can be one or more than one, such as three. Exchanging more than one pair during the move can bring balance to the cluster 128 faster than exchanging a single pair. However, performing too many moves at once may increase the query latency of the cluster, and unintentionally unbalance the cluster 128 due to, for example, adaptive replica selection, in which the query 120 is routed to a different replica shard (and therefore node) after the move.

[0048] The shard identification instructions 210 also include criteria for identifying which shards to include in the pair. For example, the criteria include a condition that a shard and its replicas are not placed on the same node and hardware rack, because doing so would reduce the redundancy and fault tolerance of the cluster 128. As another example, the criteria may include the condition that a shard corresponding to an index with active data ingestion is not moved, because doing so would disrupt the Figure 1114 depicted in . Thus, upon executing the shard identification instructions 210, the shard allocation manager 156 may disqualify shards that do not meet the criteria. For example, a hot node shard that has a replica on a cold node (or vice versa) may be disqualified from selection, as may a shard that is experiencing active data ingestion. Additionally or alternatively, the shard allocation manager 156 may initially select a first high query load shard of a hot node that receives the highest query load and a first low query load shard of a cold node that receives the lowest query load, evaluate whether the first high query load shard and the first low query load shard meet the criteria, and then confirm the selection of the first high query load shard and the first low query load shard (e.g., confirm in response to the first high query load shard and the first low query load shard meeting the criteria) or adjust the selection until a pair that meets the criteria is selected. By way of example, if the first high query load shard has a replica on a cold node, the shard allocation manager 156 may select a second high query load shard with the second highest query load on a hot node and re-evaluate.

[0049] Once the shard allocation manager 156 identifies at least one pair of shards to be swapped between a hot node and a cold node, the shard allocation manager 156 issues a command 158, which is received by the cluster controller 142 of the near real-time search platform 116. For the identified pair, the command 158 specifies which shard of the hot node is to be relocated to the cold node, and which shard of the cold node is to be relocated to the hot node. The cluster controller 142 performs the swap by the following steps: writing the identified high query load shards from the hot node to the cold node; writing the identified low query load shards from the cold node to the hot node; deleting the high query load shards from the hot node; deleting the low query load shards from the cold node. After performing the swap, the cluster controller 142 outputs a cluster state 146, which reports the current location of the shards relative to the nodes. The cluster state 146 provides an indication to the shard allocation manager 156 that the swap has been performed, for example, by indicating that the high query load shards are active on the cold node and the low query load shards are active on the hot node.

[0050] After the shard swap has occurred, in response to receiving the cluster state 146, the shard allocation manager 156 re-evaluates the cluster balance, for example by repeating the query load evaluation instruction 202 and receiving a new query load measurement 206 at the current time. Figure 5In detail, if the cluster 128 remains unbalanced, the node identification instructions 208 and the shard identification instructions 210 are repeated. Alternatively, if the cluster 128 is deemed balanced, the shard allocation manager 156 generates and outputs a shard move report 212, which is stored in the storage device 154 as part of the dynamic shard allocation log 214. For example, the shard move report 212 specifies that the dynamic shard allocation algorithm 160 was executed, which shards were moved during execution, etc. If no shards were moved, the execution of the dynamic shard allocation algorithm 160 is logged along with an indication that the cluster is sufficiently balanced.

[0051] Although the query load evaluation instructions 202, the node identification instructions 208, and the shard identification instructions 210 are described separately above, it should be understood that at least portions of the query load evaluation instructions 202, the node identification instructions 208, and the shard identification instructions 210 can be combined. In addition, it should be understood that in one or more implementations, at least portions of the query load measurement 206, the node identification instructions 208, and / or the shard identification instructions 210 are performed in parallel.

[0052] Figure 3A and Figure 3B An illustrative example 300 of dynamic shard allocation for cluster balancing in a near real-time search platform is shown. Figure 1 , the illustrative example 300 includes the cluster 128, the first node 130, the second node 132, and the third node 134. Although the illustrative example 300 includes three nodes in the cluster 128, it should be understood that more than three nodes may be included.

[0053] like Figure 3A and Figure 3BAs shown, the indexes 302 are distributed in the cluster 128, for example, in the shard 136 of the first node 130, the shard 138 of the second node 132, and the shard 140 of the third node 134. The indexes 302 are displayed according to the creation time, where the newer indexes are positioned toward the left and the older indexes are positioned toward the right. Therefore, the newest index is located at the leftmost position and the oldest index is located at the rightmost position. The fill of the index 302 and the corresponding shards represents the relative query load (e.g., during the query load measurement 206), as indicated by the query load legend 304. High query load is indicated by dark fill, medium query load is indicated by light fill, and low query load is indicated by open fill, such as no fill or white fill. In the illustrative example 300, the newer indexes 302 receive more queries than the older indexes 302. That is, the eight newest indexes 302 receive high query load, as indicated by the dark fill, and the ten oldest indexes 302 receive low query load, as indicated by the open fill. The remaining six indexes between the eight newest indexes and the ten oldest indexes receive a moderate query load, as indicated by the light fill.

[0054] exist Figure 3A In the first cluster state 306 shown, high query load indexes are concentrated in the first node 130, including in shards 308(A), shards 308(B), shards 308(C), shards 308(D), shards 308(E), and shards 308(F). The shards 136 of the first node 130 also include shards 308(G) storing medium query load indexes and shards 308(H) storing low query load indexes. The shards 138 of the second node 132 include a mix of high query load indexes, medium query load indexes, and low query load indexes. The shards 138 include shards 310(A), shards 310(B), shards 310(C), shards 310(D), shards 310(E), shards 310(F), shards 310(G), and shards 310(H). Shard 310(A) and shard 310(E) are high query load shards, while shard 310(B) and shard 310(C) are low query load shards. Shard 310(D), shard 310(F), shard 310(G), and shard 310(H) are medium query load shards. The low query load index is concentrated in the third node 134, including in shard 312(A), shard 312(B), shard 312(C), shard 312(D), shard 312(E), shard 312(F), and shard 312(G). The shards 140 of the third node 134 also include shard 312(H) storing the medium query load index.

[0055] Because high query load indexes are concentrated in the first node 130 and low query load indexes are concentrated in the third node 134, the cluster 128 is unbalanced in the first cluster state 306. For example, as depicted in the node activity chart 314, the first node 130 receives a high query load, the second node 132 receives a medium query load, and the third node 134 receives a low query load. As a result, the central processing unit (CPU) utilization of the first node 130 may approach 100%, while the third node 134 has underutilized computing resources. In addition, the transaction time of queries served by the first node 130 increases compared to the second node 132 and the third node 134. Therefore, the shard allocation manager 156 executes the dynamic shard allocation algorithm 160 (such as described above with respect to Figure 2 ), so as to more evenly distribute the query load across the cluster 128.

[0056] Now refer to Figure 3B , a second cluster state 316 is shown. In the second cluster state 316, the shards 308(A), 308(B), and 308(C) storing the high query load index have been swapped with the shards 312(A), 312(B), and 312(C) storing the low query load index. That is, the shards 308(A), 308(B), and 308(C) have been written to the third node 134 and deleted from the first node 130, and the shards 312(A), 312(B), and 312(C) have been written to the first node 130 and deleted from the third node 134. Therefore, the node activity chart 314 shows that all three nodes receive a moderate query load, indicating that the query load is substantially evenly distributed in the cluster 128. Therefore, compared to the first cluster state 306, the transaction time of the query served by the first node 130 is reduced, and computing resources are more efficiently utilized in the cluster 128.

[0057] Now refer to Figure 4 Further describe the impact of dynamic shard allocation on the performance of the near real-time search platform. Figure 4 An example of transaction time and CPU usage for two nodes before and after cluster balancing via dynamic shard allocation has been performed is shown. In particular, Figure 4 An example graph 400 of transaction time (in milliseconds (ms)) and an example graph 402 of CPU usage (in percentage) for two nodes are depicted. Graph 400 includes a dashed line graph 404 of transaction time (vertical axis) versus time (horizontal axis) for a first node and a graph 406 of transaction time versus time for a second node. Graph 402 includes a dashed line graph 408 of CPU usage (vertical axis) versus time (horizontal axis) for a first node and a graph 410 of CPU usage versus time for a second node. Figure 4In the example depicted in FIG. 4 , graph 400 and graph 402 are time aligned, and in the following discussion, reference will be made to time points of interest.

[0058] The first duration occurs between the first time point t1 and the second time point t2. During the first duration, the first node is a high query load node and the second node is a low query load node. Therefore, the transaction time of the first node is approximately twice the transaction time of the second node, as depicted by the comparison of the dotted line graph 404 and the graph 406 in the graph 400. In addition, as shown in the graph 402, the CPU utilization rate of the first node (dotted line graph 408) is close to 100% during the first duration, indicating that the resource utilization rate of the first node is at full load. In contrast, the CPU utilization rate of the second node (graph 410) is less than 20%, indicating that the second node does not fully utilize computing resources. Therefore, a query load imbalance is detected between the first node and the second node.

[0059] In response to detecting an imbalance between the first node and the second node, the dynamic shard allocation algorithm 160 is executed during a second duration between the third time point t3 and the fourth time point t4. By way of example, at least one high query load shard of the first node is exchanged with at least one low query load shard of the second node. While the dynamic shard allocation algorithm 160 is in progress, the transaction time of the first node (dashed line graph 404) decreases toward the transaction time of the second node (graph 406), particularly during the second half of the second duration. Similarly, at about half of the second duration, the CPU utilization of the first node (dashed line graph 408) decreases, while the CPU utilization of the second node (graph 410) increases.

[0060] The dynamic shard allocation algorithm 160 is executed again during the third duration between the fifth time point t5 and the sixth time point t6, such as in response to the imbalance that still exists between the first node and the second node. During the third duration, before cluster balancing is performed, the transaction time of the first node (dashed line graph 404) is about 100ms, which is about half of the transaction time during the first duration. During the third duration, the transaction time of the second node (graph 406) is similar to the transaction time of the first node, such as about 80ms. In addition, the CPU utilization rate of the first node (dashed line graph 408) and the CPU utilization rate of the second node (graph 410) are substantially the same at about 30%, as shown in the overlapped graph of the graph 402 during the third duration. Therefore, the first node and the second node are considered to be balanced relative to each other, and the query load is distributed between the first node and the second node in a substantially equal manner (e.g., within a standard deviation from the median).

[0061] By balancing the query load distribution between the first node and the second node, the average transaction time of the first node and the second node is reduced, so that the query result 148 can be returned faster and the occurrence of timeout is reduced. In addition, compared with before the balanced state, the average CPU usage between the first node and the second node is reduced, thereby achieving effective resource utilization. Overall, executing the dynamic shard allocation algorithm 160 improves the performance of the near real-time search platform 116.

[0062] Example Process

[0063] This section describes an example of a process for dynamic shard allocation in a near real-time search platform. Aspects of the process can be implemented in hardware, firmware, software, or a combination thereof. The process is illustrated as a set of boxes that specify operations performed by one or more devices and is not necessarily limited to the illustrated order for performing the operations of the individual boxes.

[0064] Figure 5 is a flow chart that depicts an algorithm as a step-by-step process 500 in an example of implementing dynamic shard allocation in a near real-time search platform. In at least one implementation, process 500 consists of Figure 1 The step-by-step process 500 may be performed by the analysis and management platform 150 of the service provider system 102 of the embodiment of the present invention. For example, the analysis and management platform 150 includes (or otherwise accesses) instructions stored in one or more computer-readable storage media, which when executed by at least one processor (or processing system) causes the at least one processor (or processing system) to perform the step-by-step process 500 as a series of operations, such as the dynamic shard allocation algorithm 160. At least a portion of the one or more computer-readable storage media and / or at least one processor may be remote from the analysis and management platform 150 and accessed via the network 108.

[0065] In one or more implementations, the step-by-step process 500 is performed according to a predetermined schedule and / or at a predetermined frequency (e.g., daily, weekly, monthly, etc.). For example, the predetermined schedule and / or the predetermined frequency may be selected based on the size and index creation pattern of the cluster 128. For example, if the index creation pattern includes creating daily indexes, the step-by-step process 500 may be performed daily to ensure that the cluster 128 remains balanced in response to the creation of new indexes.

[0066] The query load distribution of the cluster is evaluated in real time (block 502). By way of example, the analysis and management platform 150 receives a query load measurement 206 that includes the number of queries received by the cluster 128 during a predetermined time interval on a per-node and per-shard basis. For example, the predetermined time interval may be a duration in the range of one minute to three minutes, such as two minutes. In variations, the predetermined time interval is less than one minute or greater than three minutes.

[0067] A determination is made as to whether the query load distribution is balanced in the cluster (block 504). By way of example, to determine whether the query load distribution is balanced in the cluster, the shard allocation manager 156 calculates the median number of queries per node in the cluster and the standard deviation of the number of queries per node based on the query load measurements 206. When the highest number of queries per node in the cluster is within a threshold from the median number, the query load distribution is considered balanced in the cluster. In contrast, when the highest number of queries per node is at least a threshold greater than the median number, the query load distribution is not considered balanced in the cluster (e.g., the query load distribution is unbalanced). In at least one implementation, the threshold is equal to the standard deviation. In variations, the threshold is equal to a predetermined percentage of the median number.

[0068] In response to the query load distribution being balanced in the cluster, the execution of the algorithm is logged (block 506). By way of example, the execution of the dynamic shard allocation algorithm 160 is logged in the dynamic shard allocation log 214 along with an indication that the query load distribution is balanced. In one or more implementations, the analysis and management platform 150 may also output a monitoring notification 168 indicating that the dynamic shard allocation algorithm 160 was executed and the results of the algorithm (e.g., the cluster 128 is balanced and no shard moves were performed during execution).

[0069] In contrast, in response to the query load distribution being unbalanced in the cluster, hot nodes and cold nodes of the cluster are identified based on the query load distribution (block 508). By way of example, identifying the hot nodes and cold nodes of the cluster includes (e.g., by the shard allocation manager 156 executing the node identification instructions 208) sorting the nodes of the cluster 128 according to the number of queries, such as from the highest number of queries (and therefore the highest query load) in descending order to the lowest number of queries. Alternatively, the nodes may be sorted in ascending order from the lowest number of queries to the highest number of queries. In at least one implementation, the hot nodes receive the highest number of queries for the cluster 128 during the predetermined interval, and the cold nodes receive the lowest number of queries for the cluster 128 during the predetermined interval. Thus, the hot nodes receive more queries than other nodes of the cluster 128 during the predetermined interval, and the cold nodes receive fewer queries than other nodes of the cluster 128 during the predetermined interval.

[0070] At least one pair of shards to be exchanged between a hot node and a cold node is identified based at least in part on the query load distribution, the at least one pair of shards comprising a high query load shard of the hot node and a low query load shard of the cold node (block 510). By way of example, identifying at least one pair of shards to be exchanged between the hot node and the cold node comprises (e.g., by the shard allocation manager 156 executing the shard identification instructions 210) sorting the shards of the hot node and the shards of the cold node according to the number of queries (such as in ascending or descending order) so as to identify the shards that experience the highest query load in the hot node and the shards that experience the lowest query load in the cold node during a predetermined time interval. The number of pairs to be identified is specified by the shard identification instructions 210 and may be a programmable value. As a non-limiting example, the number of pairs is three. However, in variations, the number is less than three (e.g., one or two) or greater than three (e.g., in the range of four to ten). Exchanging more than one pair during the move may bring balance to the cluster 128 more quickly than when a single pair is exchanged. However, performing too many moves at once may increase query latency on the cluster.

[0071] Identifying at least one pair of shards to be swapped between a hot node and a cold node is also based on additional criteria for identifying valid moves based on, for example, configuration constraints of the cluster. For example, the additional criteria specify that shards and replicas are not placed on the same node and hardware rack because doing so would reduce the redundancy and fault tolerance of the cluster 128. The additional criteria further specify that shards that are undergoing active data ingestion are not moved because doing so would corrupt data storage. Therefore, shards that do not meet the additional criteria can be disqualified from selection, even if these shards would be selected based on their query load numbers. For example, hot node shards that have replicas on cold nodes (or vice versa) and shards that are undergoing active data ingestion can be disqualified from selection.

[0072] Additionally or alternatively, identifying at least one pair of shards to be swapped between a hot node and a cold node can be an iterative process that includes initially selecting shards of the hot node receiving the highest query load and shards of the cold node receiving the lowest query load, evaluating whether the shards meet criteria, and confirming them (e.g., confirming in response to satisfying the criteria) or adjusting the selection until a pair is selected that satisfies the criteria.

[0073] As an illustrative example where the number of pairs is three, the highest query load shard of the hot node is identified based on the shard ordering. The highest query load shard is not undergoing active data ingestion and has no replica on the cold node. Therefore, the highest query load shard is selected for the first pair. There is a replica of the second highest query load shard of the hot node on the cold node. Therefore, the second highest query load shard is disqualified from selection. The third highest query load shard of the hot node has no replica on the cold node and is not undergoing data ingestion. Therefore, the third highest query load shard of the hot node is selected for the second pair. The fourth highest query load shard of the hot node is undergoing active data ingestion and is therefore not selected. Alternatively, the fifth highest query load shard of the hot node is selected for the third pair, which fifth highest query load shard is not undergoing data ingestion and has no replica on the cold node. Similar processing is performed for the cold node, starting with the lowest query load shard of the cold node and processing toward the highest query load shard of the cold node.

[0074] It should be understood that in the event that none of the shards of the hottest node meet the criteria, a second hot node (e.g., a node experiencing the second largest query load) can be selected as the hot node, and so on. Similarly, when none of the shards of the coldest node meet the criteria, a second cold node (e.g., a node experiencing the second lowest query load) can be selected as the cold node. When more than one pair is to be selected for exchange, shards can be selected from different hot nodes and / or different cold nodes for one or more pairs. As an illustrative example, a first pair can include a shard from the hottest node and a shard from the coldest node, and in response to none of the other shards of the hottest node meeting the criteria, a second pair can include a shard from the second hot node and another shard from the coldest node.

[0075] Commanding to swap at least one pair of shards between a hot node and a cold node (block 512). By way of example, commanding to swap at least one pair of shards between a hot node and a cold node includes specifying, for at least one pair of shards, which shard of the hot node is to be relocated to the cold node and which shard of the cold node is to be relocated to the hot node. The command may be issued, for example, by the shard allocation manager 156 and received by the cluster controller 142, which then performs the swap for at least one pair of shards by: writing the high query load shard to the cold node; writing the low query load shard to the hot node; deleting the high query load shard from the hot node; deleting the low query load shard from the cold node.

[0076] Re-evaluate the query load distribution in the cluster in response to receiving an indication that a swap has occurred (block 514). By way of example, after performing the swap, cluster controller 142 outputs cluster state 146, which reports the current position of the shards relative to the nodes. Cluster state 146 provides an indication that a swap has occurred, for example, by indicating that high query load shards are active on cold nodes and low query load shards are active on hot nodes. Thus, process 500 includes receiving, by shard allocation manager 156, an indication that a swap has occurred. In response to receiving an indication that a swap has occurred (e.g., via cluster state 146), obtaining query load measurements 206 again within a predetermined time interval.

[0077] It is again determined whether the query load distribution is balanced in the cluster (block 516). By way of example, to determine whether the query load distribution is balanced in the cluster, the shard allocation manager 156 recalculates the median number of queries per node in the cluster and the standard deviation of the number of queries per node. The median number of queries per node and the standard deviation may be different from the median number of queries per node and the standard deviation determined at block 504 because not only are the calculations performed on different query load measurement 206 data, but the swapping of at least one pair of shards between hot nodes and cold nodes affects the query load distribution in the cluster. Thus, the threshold may be a different value at block 516 than at block 504, but is determined in the same manner.

[0078] If the query load distribution is balanced in the cluster, the shard moves that occurred during cluster balancing are recorded (block 518). By way of example, execution of the dynamic shard allocation algorithm 160 is recorded in the dynamic shard allocation log 214 along with an indication that the query load distribution was balanced during execution by swapping particular shards between hot nodes and cold nodes. Thus, the identities of the shards that were moved and the identities of the nodes involved in the swap are recorded. In one or more implementations, the analysis and management platform 150 may also output a monitoring notification 168 indicating that the dynamic shard allocation algorithm 160 was executed and the results of the algorithm (e.g., that the cluster 128 was balanced and which shards were moved during execution).

[0079] On the other hand, if the query load distribution is not yet balanced in the cluster, it is determined whether a threshold number of shard moves has been reached (box 520). By way of example, a shard move refers to exchanging a pair of shards between a hot node and a cold node. Therefore, if a threshold number of shard pairs have been exchanged during a single execution of the dynamic shard allocation algorithm 160 and / or within a predetermined time period in an attempt to balance the query load distribution in the cluster, the threshold number of shard moves has been reached. The threshold number of shard moves is a predefined, adjustable non-zero integer value stored in a memory. As a non-limiting example, the threshold number is in a range between 50 and 100, such as 70. In addition, the predetermined time period relates to the frequency of execution of the dynamic shard allocation algorithm 160. For example, if the dynamic shard allocation algorithm 160 is executed every day, the predetermined time period can be one day.

[0080] If the threshold number of shard moves has been reached, the shard moves that occurred during cluster balancing are logged (block 518). By way of example, execution of the dynamic shard allocation algorithm 160 is logged in the dynamic shard allocation log 214 along with an indication that the query load distribution was unbalanced during execution, as well as the identities of the shards that were moved and the identities of the nodes involved in the exchange. In one or more implementations, the analysis and management platform 150 may also output a monitoring notification 168 indicating that the dynamic shard allocation algorithm 160 was executed and the results of the algorithm (e.g., the cluster 128 was not successfully balanced during execution and which shards were moved).

[0081] If the threshold number of shard moves has not been reached, then the hot nodes and cold nodes of the cluster are identified based on the query load distribution (block 508). By way of example, one or both of the hot nodes and cold nodes may have different identities than during a previous iteration due to a previously performed shard swap that affected the query load distribution. Thus, the shard allocation manager 156 continues to perform dynamic shard allocation until the cluster 128 is deemed balanced or the threshold number of moves is reached, at which point the dynamic shard allocation algorithm 160 terminates.

[0082] Thus, the dynamic shard allocation techniques described herein enable real-time analysis of query loads to identify shard moves that are expected to bring balance to the query load distribution of the cluster 128. Notably, the dynamic shard allocation algorithm 160 does not rely on cluster-level node configurations and static criteria for classifying specific nodes as "hot," "cold," etc., which can change with changing query traffic patterns and as new indexes are created. Instead, nodes (and shards) are classified as hot and cold relative to each other based on real-time query load distribution evaluations. In addition, because each shard move affects cluster balance in a way that cannot be mathematically predicted, such as due to adaptive replica selection that affects query rates per shard and per node, cluster balance can be effectively achieved in a manner that is adaptable to a variety of use case scenarios by performing a small number (e.g., three) of shard moves at a time and re-evaluating the query load distribution in the cluster. By balancing the query load distribution in the cluster by moving previously created shards between nodes, query latency for the near real-time search platform 116 can be reduced, while computing resources are more efficiently used for overall performance enhancements of the near real-time search platform 116.

[0083] Example Systems and Devices

[0084] Figure 6 An example of a system is shown generally at 600, which includes an example of a computing device 602, which represents one or more computing systems and / or devices that may implement the various techniques described herein. This is illustrated by the inclusion of an executable service platform 110. The computing device 602 may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0085] The example computing device 602 as shown includes a processing system 604, one or more computer-readable media 606, and one or more I / O interfaces 608 that are communicatively coupled to each other. Although not shown, the computing device 602 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one of different bus structures or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus that utilizes any one of a variety of bus architectures. Various other examples are also contemplated, such as control lines and data lines.

[0086] Processing system 604 represents a function for performing one or more operations using hardware. Therefore, processing system 604 is shown as including hardware elements 610 that can be configured as processors, functional blocks, etc. This can include hardware implementations as follows: a dedicated integrated circuit or other logic device formed using one or more semiconductors. Hardware elements 610 are not limited by the materials from which they are formed or the processing mechanisms employed. For example, a processor can be composed of semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a case, the processor executable instructions can be electronic executable instructions.

[0087] Computer readable media 606 is shown to include memory / storage 612. Memory / storage 612 represents the capacity of a memory / storage associated with one or more computer readable media. Memory / storage 612 may include volatile media (e.g., random access memory (RAM)) and / or non-volatile media (e.g., read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage 612 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). As further described below, computer readable media 606 may be configured in various other ways.

[0088] Input / output interface 608 represents functionality that enables a user to input commands and information to computing device 602, and also enables information to be presented to a user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., which may employ visible wavelengths or invisible wavelengths such as infrared frequencies to identify movement as gestures that do not involve touch), etc. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a network card, a tactile response device, etc. Thus, as further described below, computing device 602 may be configured in various ways to support user interaction.

[0089] Various techniques may be described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that the techniques can be implemented on a variety of commercial computing platforms with a variety of processors.

[0090] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include a variety of media accessible by computing device 602. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0091] "Computer-readable storage media" may refer to media and / or devices that enable persistent and / or non-transient storage of information, as compared to signal transmission alone, carrier waves, or the signal itself. Therefore, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media, and / or storage devices implemented with methods or technologies suitable for storing information thereon (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to: RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible media, or an article suitable for storing the desired information and accessible by a computer.

[0092] "Computer-readable signal media" may refer to signal-bearing media that is configured to send instructions to the hardware of the computing device 602, for example, via a network. Signal media may typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, a data signal, or other transmission mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that sets or changes one or more characteristics of a signal in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0093] As previously described, hardware elements 610 and computer-readable media 606 represent modules, programmable device logic, and / or fixed device logic that can be used in some embodiments to implement at least some aspects of the technology described herein (e.g., execute one or more instructions) implemented in hardware. Hardware can include components of integrated circuits or components of systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or other hardware. In this case, the hardware can be operated as follows: a processing device that performs program tasks defined by instructions and / or logic embodied by the hardware, and hardware for storing execution instructions such as the previously described computer-readable storage medium.

[0094] The aforementioned combination may also be used to implement the various techniques described herein. Therefore, software, hardware or executable modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium, and / or implemented by one or more hardware elements 610. Computing device 602 may be configured to implement specific instructions and / or functions corresponding to software modules and / or hardware modules. Therefore, the implementation of a module that can be executed as software by computing device 602 may be implemented at least partially in hardware, for example, by using a computer-readable storage medium and / or a hardware element 610 of a processing system 604. Instructions and / or functions may be executed / operated by one or more articles (e.g., one or more computing devices 602 and / or processing systems 604) to implement the techniques, modules and examples described herein.

[0095] The techniques described herein can be supported by various configurations of computing device 602 and are not limited to the specific examples of the techniques described herein. The functionality can also be implemented in whole or in part on the "cloud" 614 using a distributed system, for example, via platform 616, as described below.

[0096] Cloud 614 includes and / or represents a platform 616 for resources 618. Platform 616 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 614. Resources 618 may include applications and / or data that may be utilized when performing computer processing on a server remote from computing device 602. Resources 618 may also include services provided over the Internet and / or over a subscriber network (e.g., a cellular or Wi-Fi network).

[0097] The platform 616 can abstract resources and functionality to connect the computing device 602 with other computing devices. The platform 616 can also be used to abstract the scale of resources to provide a corresponding level of scale to the demand encountered for resources 618 (which are implemented via the platform 616). Therefore, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 600. For example, the functionality can be implemented in part on the computing device 602 and can be implemented via the platform 616 that abstracts the functionality of the cloud 614.

[0098] in conclusion

[0099] Although the systems and techniques have been described in language specific to structural features and / or methodological acts, it should be understood that the systems and techniques defined in the appended claims are not necessarily limited to the specific features or acts described. Rather, these specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A computing system for dynamic shard allocation, comprising: A cluster, the cluster comprising a plurality of nodes, each of the plurality of nodes comprising a plurality of shards storing searchable data; one or more processors; as well as A computer-readable storage medium storing instructions that, in response to being executed by the one or more processors, cause the one or more processors to perform operations comprising: evaluating query load distribution of the cluster in real time; identifying a first shard of a first node and a second shard of a second node based on the query load distribution; and Instruct the first shard of the first node to be exchanged with the second shard of the second node.

2. The computing system of claim 1, wherein: Evaluating the query load distribution of the cluster in real time includes measuring the number of queries per node during a defined interval.

3. The computing system of claim 2, wherein: Identifying a first shard for the first node and a second shard for the second node is in response to a highest number of queries per node being at least a threshold greater than a median number of queries per node.

4. The computing system of claim 3, wherein: The threshold is equal to the standard deviation of the number of queries per node.

5. The computing system of claim 2, wherein: The first node has a highest number of queries per node within the defined interval, and the second node has a lowest number of queries per node within the defined interval.

6. The computing system of claim 1, wherein: Identifying a first shard of the first node and a second shard of the second node based on the query load distribution includes: identifying the first node based on the first node receiving more queries than other nodes of the cluster within a defined interval; identifying the first shard based on the first shard receiving more queries than other shards of the first node within the defined interval; identifying the second node based on the second node receiving fewer queries than other nodes of the cluster within the defined interval; and The second shard is identified based on a number of queries received by the second shard compared to other shards of the second node and an identity of the first shard.

7. The computing system of claim 6, wherein: Identifying the second shard based on the number of queries received by the second shard compared to other shards of the second node and the identity of the first shard includes: selecting a shard of the second node that receives the lowest number of queries and does not include a replica on the first node as the second shard.

8. The computing system of claim 1, wherein: Instructing the first shard of the first node to exchange with the second shard of the second node includes: Command to write the first shard data into the second node; commanding the second shard data to be written into the first node; and After the first shard is indicated as active on the second node and the second shard is indicated as active on the first node, commanding deletion of the first shard data from the first node and deletion of the second shard data from the second node.

9. The computing system of claim 1, wherein: The computer-readable storage medium stores further instructions that, in response to being executed by the one or more processors, cause the one or more processors to perform operations comprising: After commanding the first shard of the first node to be exchanged with the second shard of the second node, receiving an indication that the first node is active on the second node and the second shard is active on the first node; and In response to receiving the indication, the query load distribution of the cluster is re-evaluated in real time.

10. A method for dynamic shard allocation, comprising: evaluating a query load distribution of a cluster of nodes over a predetermined interval, the nodes of the cluster of nodes including shards storing searchable data; identifying, based on the query load distribution, at least one pair of shards to be exchanged between a high query load node of the node cluster and a low query load node of the node cluster; exchanging the at least one pair of shards between the high query load node and the low query load node; as well as In response to receiving an indication that the at least one pair of shards has been swapped between the high query load node and the low query load node, re-evaluating the query load distribution of the node cluster in real time.

11. The method according to claim 10, wherein: Identifying at least one pair of shards to be exchanged between a high query load node of the node cluster and a low query load node of the node cluster based on the query load distribution is in response to a highest number of queries per node being greater than a median number of queries per node during the predetermined interval by at least a threshold.

12. The method according to claim 10, wherein: The high query load node receives a highest number of queries per node during the predetermined interval, and the low query load node receives a lowest number of queries per node during the predetermined interval.

13. The method according to claim 10, wherein: The at least one pair of shards includes a high query load shard of the high query load node and a low query load shard of the low query load node.

14. The method according to claim 13, wherein: The high query load shard is the highest query load shard of the high query load node excluding the first replica shard on the low query load node, and the low query load shard is the lowest query load shard of the low query load node excluding the second replica shard on the high query load node.

15. The method according to claim 13, wherein: The high query load shard and the low query load shard do not actively ingest new data for storage.

16. A method for dynamic shard allocation, comprising: evaluating, in real time, a query load distribution of a cluster, the cluster comprising a plurality of nodes, each node comprising a shard storing searchable data; determining whether the cluster is balanced based on the query load distribution; In response to the cluster being unbalanced: identifying high query load nodes and low query load nodes among the plurality of nodes; identifying a high query load shard of the high query load node and a low query load shard of the low query load node based on the query load distribution; as well as Exchanging the high query load shard of the high query load node with the low query load shard of the low query load node; as well as In response to the cluster being balanced, an indication of the cluster being balanced is generated.

17. The method according to claim 16, wherein: Identifying the high query load nodes and the low query load nodes among the plurality of nodes is also in response to a threshold number of shard moves not being reached within a time period.

18. The method according to claim 17, further comprising: In response to reaching a threshold number of shard moves within the time period, generating a second indication that the cluster is unbalanced without exchanging additional shards between the plurality of nodes.

19. The method according to claim 16, wherein: Identifying the high query load shards of the high query load nodes and the low query load shards of the low query load nodes is also based on configuration constraints of the cluster.

20. The method according to claim 19, wherein: The configuration constraints of the cluster include that the high query load shard of the high query load node does not have a first shard copy stored on the low query load node, and the low query load shard of the low query load node does not have a second shard copy stored on the high query load node.

Images