A method for scaling up nodes in a distributed search engine

Abstract

This application discloses a method for scaling up nodes in a distributed search engine, including obtaining the current information of the distributed search engine cluster and the information of the scaling nodes; calculating the total number M of shards that the scaling nodes can accommodate based on the current information of the cluster and the information of the scaling nodes; adding the scaling nodes to the cluster; performing multiple rounds of node pre-allocation, each round of node pre-allocation including: establishing a weight index and dividing the weight index into N weight shards; performing data balancing on the cluster; and completing node allocation when N reaches a preset value. The solution provided by this application solves the problem in the prior art where distributed search engines cannot scale up the cluster in-situ or on demand when the cluster data capacity or data write performance reaches its limit, resulting in resource waste.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method and apparatus for expanding the node capacity of a distributed search engine. This application further provides related electronic devices and non-volatile computer-readable storage media. Background Technology

[0002] A distributed search engine breaks down a large amount of index data into multiple chunks, distributing the data across multiple independent servers. These servers then perform searches on the distributed data, forming a complete distributed architecture. Similarly, a distributed network storage system employs a scalable architecture, using multiple storage servers to share the storage load and location servers to locate stored information, thereby improving system reliability, availability, and access efficiency.

[0003] Elasticsearch (ES) is a typical distributed search engine. It is a distributed, multi-tenant, full-text open-source search engine based on Apache Lucene. It has a simple and easy-to-use HTTP web query and management interface and supports fast search and analysis of all types of data, including text, numbers, geospatial, structured and unstructured data. Therefore, ES has a wide range of applications in the field of log retrieval.

[0004] Normally, Elasticsearch (ES) clusters have limited data capacity and performance. However, when data demand surges, such as during product promotions, the data volume can easily increase dramatically. When the ES cluster's data capacity or the amount of data written reaches its limit, data cannot be written in a timely manner, impacting log search services. This necessitates adding new processing nodes. Due to Elasticsearch's underlying sharding mechanism, when a new node is added, data is sharded for distribution. This causes the cluster's workload to shift to the new node, resulting in excessive data load and impacting data processing performance. Furthermore, because the new node experiences concentrated data writing over a period, it can lead to severe hotspot concentration, causing the entire cluster's workload to be primarily borne by the new node, resulting in uneven workload distribution and a significant discrepancy between actual and potential processing capabilities.

[0005] In existing technologies, there are two main approaches to scaling the data capacity and performance of Elasticsearch (ES) clusters: one is to plan capacity in advance and establish a large-scale ES cluster; the other is to rebuild an even larger ES cluster, with new data written directly to the new cluster and old data imported from the old cluster back to the new cluster. Both approaches have shortcomings. The problem with planning capacity in advance and establishing a large-scale ES cluster is that pre-estimating capacity is often difficult, and if a large cluster is established beforehand, low cluster utilization may occur, leading to resource waste. The problem with rebuilding an even larger ES cluster is that log querying may become fragmented during data migration, requiring data to be queried from both the old and new clusters and then integrated. Furthermore, data re-importation may impact the writing of new data. Clearly, neither of these methods can achieve in-situ scaling of a distributed search engine and is prone to resource waste. Therefore, neither method is an effective solution to the technical shortcomings of scaling ES distributed search engines. These technical problems that need to be addressed may exist in various distributed search engines. Summary of the Invention

[0006] In view of this, this application provides a method and apparatus for expanding the node capacity of a distributed search engine. This application further provides related electronic devices and computer-readable storage media. The above-mentioned technical solutions provided by this application aim to solve the problems in the prior art that when the cluster data capacity or data writing performance of the distributed search engine reaches its limit, it is impossible to expand the distributed search engine cluster in place or to expand it on demand, resulting in resource waste.

[0007] This application provides a method for scaling up nodes in a distributed search engine, including:

[0008] Obtain the current information of the distributed search engine cluster and the information of the expansion nodes;

[0009] Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated.

[0010] Add the expansion node to the cluster;

[0011] Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index; and performing data balancing on the cluster.

[0012] Wherein, N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round;

[0013] When N reaches the preset value, the node allocation is completed.

[0014] Optionally, before adding the expansion node to the cluster, the method further includes: disabling the data balancing function of the cluster.

[0015] Optionally, before performing multiple rounds of node pre-allocation, the method further includes: creating a globally matching index template and configuring the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the nodes that cannot be allocated to the data shards corresponding to the index established based on the index template.

[0016] Optionally, after establishing the weight index in each round and before performing data balancing on the cluster, the method further includes:

[0017] Configure the second target parameter of the index template to be empty.

[0018] Optionally, after the node allocation is completed, the method further includes: deleting the globally matching index template.

[0019] Optionally, the current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes.

[0020] The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes:

[0021] Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state.

[0022] Based on the average number of data shards and the number of expansion nodes, the total number M of shards that the expansion nodes can accommodate is calculated.

[0023] Optionally, the method further includes: in each round of node pre-allocation steps, before the step of establishing the weight index, disabling the data balancing function of the cluster.

[0024] Optionally, the method further includes: in each round of node pre-allocation step, before the step of establishing the weight index, deleting the weight index established in the previous round of node pre-allocation.

[0025] Optionally, the node pre-allocation in each round further includes:

[0026] The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

[0027] This application also provides a device for scaling up nodes in a distributed search engine, comprising:

[0028] The information acquisition unit acquires the current information of the distributed search engine cluster and the information of the expansion nodes;

[0029] The sharding calculation unit calculates the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node.

[0030] The node addition unit adds the expansion node to the cluster;

[0031] The data sharding allocation unit performs multiple rounds of node pre-allocation. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight shards, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight shards are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight shards corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round; when N reaches a preset value, the node allocation is completed.

[0032] Optionally, before adding the expansion node to the cluster, the method further includes: disabling the data balancing function of the cluster.

[0033] Optionally, before performing multiple rounds of node pre-allocation, the method further includes: creating a globally matching index template and configuring the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the nodes that cannot be allocated to the data shards corresponding to the index established based on the index template.

[0034] Optionally, after establishing the weight index in each round and before performing data balancing on the cluster, the method further includes:

[0035] Configure the second target parameter of the index template to be empty.

[0036] Optionally, after the node allocation is completed, the method further includes: deleting the globally matching index template.

[0037] Optionally, the current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes.

[0038] The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes:

[0039] Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state.

[0040] Based on the average number of data shards and the number of expansion nodes, the total number M of shards that the expansion nodes can accommodate is calculated.

[0041] Optionally, the method further includes: in each round of node pre-allocation steps, before the step of establishing the weight index, disabling the data balancing function of the cluster.

[0042] Optionally, the method further includes: in each round of node pre-allocation step, before the step of establishing the weight index, deleting the weight index established in the previous round of node pre-allocation.

[0043] Optionally, the node pre-allocation in each round further includes:

[0044] The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

[0045] This application also provides an electronic device, including:

[0046] processor,

[0047] After storing the program for the node expansion method of the distributed search engine in the memory, the following steps are executed:

[0048] Obtain the current information of the distributed search engine cluster and the information of the expansion nodes;

[0049] Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated.

[0050] Add the expansion node to the cluster;

[0051] Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round; when N reaches a preset value, the node allocation is completed.

[0052] This application also provides a non-volatile computer-readable storage medium.

[0053] The non-volatile storage medium stores a program for a node expansion method of a distributed search engine, which is executed by a processor to perform the following steps:

[0054] Obtain the current information of the distributed search engine cluster and the information of the expansion nodes;

[0055] Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated.

[0056] Add the expansion node to the cluster;

[0057] Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round; when N reaches a preset value, the node allocation is completed.

[0058] Compared with the prior art, the embodiments of this application have the following advantages:

[0059] This application provides a method for scaling up nodes in a distributed search engine, including obtaining current information of the distributed search engine cluster and information of the scaling nodes; calculating the total number M of shards that the scaling nodes can accommodate based on the current information of the cluster and the information of the scaling nodes; adding the scaling nodes to the cluster; performing multiple rounds of node pre-allocation, each round of node pre-allocation including: establishing a weight index, dividing the weight index into N weight shards, and configuring a first target parameter of the weight index as the node identifier corresponding to the scaling node, so that the N weight shards are allocated to the scaling nodes, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight shards corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to each round of node pre-allocation is smaller than the N corresponding to the previous round of node pre-allocation; when N reaches a preset value, the node allocation is completed. The method provided in this application, within a distributed search engine, adds expansion nodes to the cluster, performs multiple rounds of node pre-allocation, and balances cluster data after each round of pre-allocation. This phased approach achieves cluster data balancing, enabling in-situ and on-demand expansion when the cluster's data capacity or write performance reaches its limit, thus avoiding resource waste. Furthermore, this method prevents data writing to expansion nodes from impacting write performance within a concentrated timeframe. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 It is a distributed architecture where three servers jointly manage the data;

[0062] Figure 2 This is a flowchart of a node expansion method for a distributed search engine provided in an embodiment of this application;

[0063] Figure 3 This is a flowchart illustrating the allocation of data fragments to expansion nodes according to an embodiment of this application;

[0064] Figure 4 This is an example diagram of node expansion for a distributed search engine provided in one embodiment of this application;

[0065] Figure 5 This is a block diagram of a node expansion device for a distributed search engine provided in one embodiment of this application;

[0066] Figure 6 This is a schematic diagram of the logical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0068] It should be noted that the terms "first," "second," and "third," etc., used in the various parts of the embodiments and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Such terms can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0069] This application provides an embodiment of a method for scaling up nodes in a distributed search engine. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0070] Before introducing the node scaling method of the distributed search engine provided in this application, let's first introduce the relevant background of this scheme.

[0071] Search refers to retrieving all data containing a given search term. For example, a user enters a term in the search box, the client software sends a request to the backend, and the backend uses Structured Query Language (SQL) statements to retrieve relevant entries from the database (the database compares each entry one by one). This is a very simple prototype of a search. Structured Query Language is a database query and programming language used to access, query, update, and manage relational database systems. SQL statements are a language used to manipulate the database.

[0072] However, regular searches face a problem: when dealing with large amounts of data, search efficiency becomes extremely low. From a user's perspective, it takes a long time from clicking the search button to seeing the results. With 500GB of data, this could take 1-2 hours, which is unacceptable for users. When the data volume exceeds the capacity of a single server, it becomes unusable, requiring multiple servers. Since the data is distributed across different servers, a client cannot request every single server during a search. Therefore, an administrator role is needed to distribute client requests to each server and aggregate the results before returning them to the client.

[0073] When the data volume exceeds the current server's capacity, server expansion is necessary. Generally, there are two expansion methods: vertical expansion (vertical scaling) and horizontal expansion (lateral scaling). Vertical expansion involves purchasing more powerful servers. For example, if the current server can handle 1TB of data, and the data volume reaches 3TB, a 3TB server is purchased to replace it. Horizontal expansion involves purchasing more servers to create a cluster with multiple servers of similar performance. Figure 1 The diagram illustrates a distributed architecture where three servers jointly manage data. If the current servers can handle 1TB of data, and the data volume reaches 3TB, three more servers are purchased. Each server deploys an Elasticsearch (ES) process to manage the 1TB of data on that server. This distributed search engine breaks down the large amount of index data into multiple chunks, placing a portion on each server. The search is then performed across these multiple servers, forming a complete distributed architecture.

[0074] Vertical scaling is limited, and in terms of cost, the price difference between servers with small memory capacity and relatively low performance and servers with large memory capacity and high performance is not comparable. True scaling capability comes from horizontal scaling, which involves adding more nodes to the cluster and distributing the load and stability across these nodes.

[0075] For most databases, significant application modifications are typically required to leverage the increased resources from horizontal scaling. However, distributed search engines, such as Elasticsearch, are inherently distributed and leverage multiple nodes to improve scalability and availability. A running Elasticsearch instance is called a node, and a cluster consists of one or more nodes configured with the same cluster name, sharing the load and data pressure. When a node joins or leaves the cluster, the cluster redistributes all the data; this process is called cluster data balancing.

[0076] This application uses the Elasticsearch search engine as an example, and the technical solution provided in this application can also be used in other distributed search engines.

[0077] Elasticsearch is an open-source, distributed full-text search engine built on Lucene. It features a user-friendly HTTP web query and management interface and supports fast searching and analysis of all types of data, including text, numbers, geospatial data, and both structured and unstructured data. Lucene, on the other hand, is an open-source full-text search engine toolkit; it is not a complete search engine, but rather an architecture for a full-text search engine.

[0078] A cluster consists of one or more nodes that collectively hold the entire dataset and work together to provide indexing and search capabilities. A cluster is identified by a unique name, which defaults to "elasticsearch"; a node can only join a cluster by specifying its name.

[0079] In general, from a hardware perspective, a node is an independent server in a cluster. Of course, it can also be a specific partition of a server, or even different nodes deployed on different partitions of a single server.

[0080] As part of a cluster, nodes store data and participate in the cluster's indexing and search functions. Similar to a cluster, a node is identified by a name (randomly assigned by default). Node names are important for operational management because they identify which servers in the network correspond to which nodes in the Elasticsearch cluster. A node joins a specified cluster by configuring a cluster name. By default, each node is assigned to a cluster named "elasticsearch." This means that if you start several nodes in your network and assume they can discover each other, they will automatically form and join a cluster named "elasticsearch." If no Elasticsearch nodes are currently running in your network, starting a node will create and join a cluster named "elasticsearch" by default. A cluster can contain any number of nodes, and a single node can also form an Elasticsearch cluster.

[0081] An index can be simply understood as a database, a collection of documents with some similar characteristics. For example, there could be an index for customer data, a product category index, and an order data index. An index contains many documents, representing a category of similar or identical documents. If a product index is created, it will store all product data, i.e., all product documents. An index is identified by a name, which is used when indexing, searching, updating, and deleting documents corresponding to that index. Any number of indexes can be defined within a cluster.

[0082] Shards are used to address the issue of indexes containing excessive amounts of data beyond the hardware limitations of a single node. For example, an index with 1 billion documents might occupy 1TB of disk space, which no single node could possibly handle. Even if such a large amount of disk space were available, a single node would be too slow to process search requests, resulting in inefficient search performance. To solve this problem, the Elasticsearch search engine provides the ability to divide an index into multiple parts, called shards. In practice, an index is split into multiple shards and distributed across multiple servers for storage. Sharding allows for horizontal scaling, storing more data, and distributing search and analysis operations across multiple servers, improving throughput and performance. When creating an index, the desired number of shards can be specified. Each shard is also a fully functional and independent "index" that can be placed on any node in the cluster.

[0083] A replica is a copy of a shard. Shards are divided into primary shards and replica shards. Data in an index is physically distributed across multiple primary shards, with each primary shard storing only a portion of the data. Each primary shard can have multiple replicas, which are copies of the primary shard. Replica shards act as backups, preventing damage from primary shard failures. If a shard on a node is damaged or lost, it can be recovered from the replica. They also handle query requests, ensuring a more even distribution of requests between primary and replica shards.

[0084] In existing technologies, there are two main solutions to address the scaling of data capacity and performance of Elasticsearch (ES) clusters: one is to plan capacity in advance and establish a large-scale ES cluster; the other is to rebuild an even larger ES cluster, with new data written directly to the new cluster and old data imported from the old cluster back to the new cluster.

[0085] Both existing solutions have shortcomings. The problem with the solution of planning capacity in advance and building a large-scale Elasticsearch cluster is that it is often difficult to estimate capacity beforehand. If a large-scale cluster is built in advance, the cluster utilization may be low, resulting in wasted resources. The problem with the solution of rebuilding an even larger Elasticsearch cluster is that log querying will be fragmented during data migration, requiring data to be queried from both the old and new clusters and then integrated. At the same time, re-importing data may have a certain impact on the writing of new data.

[0086] Meanwhile, if the cluster is directly expanded by adding nodes, Elasticsearch's underlying sharding mechanism will cause a large number of newly created shards to be allocated to the new nodes. This results in a concentrated migration of data to the new nodes, severely impacting data write performance. Specifically, this manifests in two ways: First, for existing indexes, as data sharding begins within the cluster, some existing shards migrate to the new nodes, leading to a decrease in read / write performance for some frequently accessed shards. Second, for new indexes, all new data shards will be allocated to the new nodes, resulting in an excessively large increase in shard size on the new nodes, making it difficult to match processing capacity and impacting data processing performance.

[0087] In general, the current method of directly adding new nodes to the cluster to achieve capacity expansion is not feasible. Figure 2 This is a flowchart of a node expansion method for a distributed search engine provided in the first embodiment of this application. The following is in conjunction with... Figure 2 This embodiment provides a detailed description of a method for scaling up nodes in a distributed search engine. The embodiments described below are for illustrative purposes only and are not intended to limit actual use.

[0088] like Figure 2 As shown in the figure, the method for scaling up nodes in a distributed search engine provided in this embodiment includes:

[0089] Step S101: Obtain the current information of the distributed search engine cluster and the information of the expansion nodes.

[0090] This step involves obtaining the current information of the distributed search engine cluster and the information of the expansion nodes. This information is used to collect relevant information before expanding the nodes, so as to determine the data that needs to be set for each subsequent step.

[0091] S101-1, through the scheduling system, periodically obtains the current information of the cluster from the ES cluster interface and stores it in external storage. The current information of the cluster includes the current number of nodes in the cluster and the current number of data shards in the cluster.

[0092] Monitoring Elasticsearch (ES) cluster information enables data developers to monitor the current status of the cluster in real time. For example, they can use "status" to understand whether the cluster is in a healthy and operational state, "number_of_nodes" to get the current number of nodes in the cluster, "number_of_data_nodes" to get the current number of data nodes in the cluster, and "active_primary_shards" to get the total number of primary shards for all indexes in the cluster.

[0093] In this step, at least the current number of nodes and the current number of data shards in the cluster are required. The current number of data shards is the total number of primary shards for all indexes in the current cluster. This can be obtained by entering "number_of_nodes" in the Elasticsearch monitoring script and "active_primary_shards" to obtain the total number of primary shards for all indexes in the cluster, i.e., the current number of data shards. The current number of nodes can be labeled as X_nodes, and the current number of data shards can be labeled as N_shards.

[0094] S101-2, Obtain information on the current cluster that needs to be expanded, including the number of nodes that need to be expanded, i.e., the number of nodes to be expanded.

[0095] When the Elasticsearch cluster reaches its data capacity or write limit, new data cannot be written in a timely manner, limiting cluster performance and impacting Elasticsearch's log search service. In this case, the cluster needs to be scaled up. The key information for scaling up the current cluster includes the number of nodes to be added, i.e., the number of nodes to be added. This number can be denoted as Y_nodes. The number of nodes to be added can be determined based on the received data processing requirements; however, the specific method for determining these requirements is not limited here.

[0096] Step S102: Calculate the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node.

[0097] This step calculates the total number of shards M that the expansion node can accommodate, based on the current information of the cluster and the information of the expansion node. The current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes. This step is used to determine the total number of shards that can be accommodated in the expansion node, so as to provide target data for subsequent processing steps.

[0098] S102-1, Calculate the average number of data shards per node when the distributed search engine is in equilibrium, based on the current number of nodes and the current number of data shards in the cluster.

[0099] The average number of data shards per node is the current number of data shards in the cluster under balanced conditions divided by the current number of nodes. The average number of data shards per node can be denoted as avg_shards, i.e., avg_shards = N_shards / X_nodes.

[0100] For example, when a cluster is in a balanced state, the current number of nodes is 2 and the current number of data shards is 10. Therefore, the average number of data shards per node is 5, meaning that in a balanced state, each node in the current cluster has 5 data shards.

[0101] S102-2, Calculate the total number of shards M that the expansion node can accommodate based on the average number of data shards per node when the cluster is in a balanced state and the number of expansion nodes.

[0102] The total number of shards M that the expansion node can accommodate is the product of the average number of data shards per node and the number of expansion nodes in the current equilibrium state, i.e., the total number of shards M that the expansion node can accommodate = avg_shards * Y_nodes. The calculated total number of shards that the expansion node can accommodate provides the basis for the number of weight shards selected for pre-allocation to the weight index in each subsequent step. The number of weight shards selected for pre-allocation in each subsequent round is determined jointly based on the total number of shards that the expansion node can accommodate and the actual production needs. In actual production, the number of weight shards that can be added is less than the total number of shards that the expansion node can accommodate.

[0103] For example, in the cluster described above, due to limited data write performance, three expansion nodes are added. The total number of shards M that these expansion nodes can accommodate is the average number of data shards per node when the cluster is in a balanced state, multiplied by the number of expansion nodes, which is 5 * 3 = 15. That is, the total number of shards M that the expansion nodes can accommodate is 15, meaning the total number of shards that can be accommodated on the expansion nodes of the current cluster is 15. This calculated value of 15 provides the basis for the number of weight shards pre-allocated to the weight index in each subsequent round of node pre-allocation.

[0104] Step S103: Add the expansion node to the cluster.

[0105] This step adds the expansion node to the cluster, that is, adds a new node, i.e., an expansion node, to the current cluster. The implementation method of this step is different for different search engines. It should be noted that after adding the expansion node, data shards cannot be stored in it immediately, but need to be added gradually according to subsequent steps.

[0106] Different distributed search engines may automatically perform different operations after adding new expansion nodes. Many of these operations may affect the balanced distribution of data. Therefore, in most cases, adding new expansion nodes requires prior configuration.

[0107] For example, with the Elasticsearch search engine, by default, when a new node is added to the cluster, the cluster redistributes all the data and immediately performs cluster data balancing. Therefore, cluster data balancing can be disabled before adding expansion nodes to the cluster to avoid this immediate execution of the cluster data balancing function after a new node is added.

[0108] The following uses specific settings parameters in the ES search engine as an example to explain the process and meaning of the above settings.

[0109] In the Elasticsearch search engine, there is a configuration option `cluster.routing.rebalance.enable`. Setting this option allows you to enable or disable rebalancing for specific types of shards. When this option is set to `all-` (the default), shard balancing is allowed for all types of shards; when set to `primaries-`, shard balancing is only allowed for primary shards; when set to `replicas-`, shard balancing is only allowed for replica shards; and when set to `none-`, no shard balancing is allowed for any index. Therefore, for the Elasticsearch search engine, before adding the scaling nodes to the cluster in this step, setting `cluster.routing.rebalance.enable` to `none` will disable shard balancing for any index, effectively turning off the cluster data balancing function.

[0110] In the preferred solution, in the actual production environment of the Elasticsearch search engine, after disabling the cluster data balancing function, in order to eliminate interference from the online official index and prevent the data shards of the newly created index from being assigned to the expansion node, it is necessary to create a globally matching index template, set the loading order value to the maximum, and add the list of node names of the nodes to be expanded to the exclusion list for cluster data writing. That is, the data shards corresponding to the index created based on the index template must not be assigned to the nodes to be expanded.

[0111] The index template is a collection of index configurations, each template having a numerical sequence identifier; the larger the sequence identifier, the higher the priority. An index can have multiple templates, with configurations overriding each other in ascending order of template identifiers. Here, a new globally matching index template `Temp` is created, and its loading order value is set to the maximum loading order value + 1, i.e., `order_max + 1`. By loading the maximum order value for this index template and adjusting its priority to the highest, the performance of scaling nodes is prevented from being affected by a continuous influx of data from newly created index shards in a production environment.

[0112] After creating a new index template `Temp`, configure the second target parameter of the index template as the node identifier corresponding to the expansion node. The second target parameter represents the nodes to which data shards corresponding to the index created based on the index template should not be assigned. Specifically, the configuration item for the second target parameter is `index.routing.allocation.exclude._name`, which is set to the node identifier of the expansion node. This can be a list of node names for the expansion node. This prevents the allocation of data shards corresponding to newly created online indexes to expansion nodes, thus preventing data shards corresponding to newly created online production indexes from entering expansion nodes and impacting cluster performance.

[0113] If the cluster data balancing function is disabled and no new globally matching index template is created or its subsequent configuration is performed, after the expansion node is added to the current cluster, a large amount of data on the data shards corresponding to the newly created online official index will be written to the newly added expansion node in a short period of time, impacting cluster performance. Implementing the creation of a new globally matching index template and subsequent operations in this step is to provide a good environment for each subsequent round of node pre-allocation, preventing new data from being written during node pre-allocation. After completing the above preparation steps, add the node to be expanded to the current cluster. When adding the expansion node to the current cluster, first ensure that the cluster name of the expansion node is consistent with the name of the cluster to be joined. When a consistent name is set, automatic discovery is enabled, and the expansion node can be automatically discovered by the current cluster. Simultaneously, the name of the expansion node should be different from the names of existing nodes in the current cluster; duplicate names are not allowed.

[0114] For example, in a balanced cluster, if there are two nodes named node1 and node2, then when setting the names of the expansion nodes, they can be named node3, node4, and node5 respectively. These three nodes will then be added to the current cluster. Of course, it must first be ensured that the cluster names of the expansion nodes {node3, node4, node5} are consistent with the name of the current cluster.

[0115] At this point, the expansion nodes are added to the cluster. Because cluster data balancing is disabled, data in the cluster will not be written to the expansion nodes at this time. In other words, the cluster already contains these expansion nodes, but these nodes will not write data. Disabling cluster data balancing and adding the expansion nodes to the current cluster prepares for subsequent step-by-step node pre-allocation and data balancing.

[0116] Disabling cluster data balancing prevents data shards from indexes on other nodes in the cluster from being allocated to the scaling node. Creating a globally matching index template and subsequent steps prevent data shards corresponding to newly created online production indexes from being allocated to the scaling node. The preferred solution provides a robust production environment, preventing data shards from other nodes in the cluster and data shards from newly added indexes from being allocated to the scaling node.

[0117] Step S104 involves performing multiple rounds of node pre-allocation. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node. The first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index. Data balancing is performed on the cluster. N is a positive integer less than M, and the N corresponding to each round of node pre-allocation is smaller than the N corresponding to the previous round of node pre-allocation. When N reaches a preset value, node allocation is completed.

[0118] This step provides a method for allocating weight shards to expansion nodes. The core of this method is to allocate weight shards to the weight index in rounds by using a series of intervals to balance the cluster data and then distribute the data shards to the expansion nodes in rounds. In this way, the problem of writing new expansion nodes into the data center will not occur during the node expansion process.

[0119] Undoubtedly, the above steps will require different implementation methods in different distributed search engines. The following will mainly introduce the specific implementation methods using the ES search engine.

[0120] In the Elasticsearch search engine, each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight shards, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight shards are allocated to the expansion node. The first target parameter is used to characterize the nodes allowed to allocate the weight shards corresponding to the weight index. Data balancing is performed on the cluster. N is a positive integer less than M, and the N corresponding to each round of node pre-allocation is smaller than the N corresponding to the previous round. When N reaches a preset value, node allocation is completed. Simultaneously, during each round of node pre-allocation, to ensure that the number of weight shards corresponding to the weight index is unrestricted, the third target parameter of the weight index can also be configured as unlimited. The third target parameter configuration characterizes the maximum number of shards that each index can allocate to each node.

[0121] In each round of node pre-allocation, before establishing the weight index, the weight index established in the previous round of node pre-allocation needs to be deleted. Because the weight slice corresponding to the weight index has characteristics set when the weight index is established, deleting the weight index also deletes the weight slices established according to the weight index. Weight slices are data slices with empty values; in each round of node pre-allocation, they only serve as placeholders, preventing other data slices from being allocated to the positions occupied by the weight slices.

[0122] Meanwhile, in each round of node pre-allocation, before establishing the weight index, the cluster's data balancing function needs to be disabled. This prevents data shards from being allocated to expansion nodes in the next round of pre-allocation, thus impacting cluster performance.

[0123] A weight index is established, which is divided into N weight segments. The first target parameter of the weight index is configured as the node identifier corresponding to the expansion node. The first target parameter is used to characterize the nodes that allow the weight segments corresponding to the weight index to be allocated. That is, the first target parameter configuration item is index.routing.allocation.include._name. Set index.routing.allocation.include._name to the name list of expansion nodes so that these N weight segments are allocated to expansion nodes. Meanwhile, during each round of node pre-allocation, the third target parameter of the weight index can be configured to be unlimited. The third target parameter configuration is used to characterize the maximum number of shards that each index can allocate to each node. That is, the third target parameter configuration item is index.routing.allocation.total_shards_per_node. The third target parameter configuration item index.routing.allocation.total_shards_per_node can be set to a negative number, so that the weight shards of each node can be infinitely large, ensuring that the number of weight shards that each expansion node can allocate is unlimited, and preventing excess weight shards from having no nodes to allocate to.

[0124] In the preferred scheme considering the actual production environment as described above, when creating a new globally matching index template, after establishing the weight index during node pre-allocation, the second target parameter of the weight index needs to be configured to be empty. That is, the data shards corresponding to the index established based on the index template cannot be allocated to nodes, even if the previously configured second target parameter is the node identifier corresponding to the expansion node, which is invalid.

[0125] After multiple rounds of node pre-allocation, when N reaches the preset value and the node allocation is completed, the globally matching index template is deleted.

[0126] In the example above, the total number of shards that the expansion node can accommodate is calculated to be 15. This means that the number of shards pre-allocated in each round cannot exceed 15. In a specific instance, the number of shards pre-allocated in the first round can be set to 12, meaning these 12 shards are allocated to the 3 expansion nodes. These 12 shards occupy 12 positions, leaving 3 empty positions. These 12 shards are then allocated to the 3 expansion nodes. After allocating these 12 shards, the cluster data balancing function is enabled. Data shards with data will then be allocated to the empty positions. The first round of node pre-allocation is now complete, and data shards with data are allocated to the expansion nodes. A second round of node pre-allocation can then be performed. The number of shards pre-allocated in the second round is set according to actual production needs. It should be noted that the number of shards created based on the weight index in the second round is less than the number created in the first round. Ten weight fragments can be created. These ten fragments are then distributed across the twelve empty slots of the three expanded nodes, leaving two empty slots. These ten fragments are then distributed across the three expanded nodes. After distributing the ten fragments, the cluster data balancing function is activated again, distributing the data fragments with data to the two empty slots. Node allocation is complete when N reaches the preset value.

[0127] It should be noted that the preset values ​​can be set according to actual production needs, and the weight shards of the selected weight index in each round can also be set, as long as the N corresponding to the node pre-allocation in each round is smaller than the N pre-allocated in the previous round. Of course, in order to steadily achieve the performance of in-situ scaling of the cluster, the set N can be regular, and the preset value can be the value that completely allocates data shards with data to the scaling nodes in the last round.

[0128] Preferably, in S104, the number of preset weight slices in each round can be regular and steadily decrease, so that the data in the data slices on the newly added expansion node are all real data.

[0129] Figure 3This document illustrates a flowchart illustrating the allocation of real data shards to expansion nodes according to a preferred embodiment of the first embodiment of this application. The allocation of data shards with empty values, i.e., weight shards, to expansion nodes includes: establishing a weight index and setting the corresponding number of weight shards, wherein the number of weight shards is set according to the following rule: the number of weight shards allocated in each round decreases by K compared to the previous round, and the number of weight shards allocated in the first round is MK; configuring a first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the weight shards in each round are allocated to the expansion node, wherein the first target parameter is used to characterize the node that allows the allocation of weight shards corresponding to the weight index; and performing data balancing on the cluster to allocate data shards containing real data to the expansion nodes.

[0130] S104-1, Establish a weight index, configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, i.e., configuring it as the list of node names for the expansion node. Establish a new weight index. The weight slice corresponding to the newly established weight index has no data; it is a data slice with empty values. This weight index and its weight slices act as "placeholders" in the expansion node, i.e., allocating the weight slice corresponding to this weight index to the newly added expansion node, restricting other data slices from being allocated to this node.

[0131] Of course, different distributed search engines can use different methods to build weight indexes, and there are no restrictions on which method is used to build weight indexes.

[0132] In the Elasticsearch search engine, creating a weight index involves sending a request to the system to configure the index. The first target parameter of the weight index is the node identifier corresponding to the expansion node. The configuration item `index.routing.allocation.include._name` is set to a list of expansion node names to complete the weight index configuration. Setting `index.routing.allocation.include._name` to a list of expansion node names allows the weight fragments corresponding to the weight index to be allocated to these expansion nodes. In the example above, the list of expansion node names could be {node3, node4, node5}.

[0133] Since this embodiment requires adding weight slices in batches, the relevant configuration for creating a new index needs to be completed by sending the above-mentioned request in each step.

[0134] S104-2, establish the weight segments of the weight index; wherein, the number of weight segments established is set according to the following rule: the number of weight segments allocated in each round is reduced by K compared with the previous round, and the number of weight segments allocated in the first round is MK.

[0135] This step is used to establish the weight segments of the weight index; wherein, the number of weight segments established is set according to the following rule: the number of weight segments allocated in each round is reduced by K compared with the previous round, and the number of weight segments allocated in the first round is MK.

[0136] Before setting the number of shards for the weight index as follows, the third target parameter needs to be configured to be unlimited, meaning the maximum number of shards that each index can allocate to each node is set to unlimited. Specifically, in the Elasticsearch search engine, the configuration item `index.routing.allocation.total_shards_per_node` can be set to a negative number, allowing each index to allocate an infinite number of shards per node. This ensures that the number of shards that can be allocated to each expansion node is unlimited, preventing excess shards from having no nodes to allocate to them.

[0137] For example, consider a system with one index, two nodes, and ten shards. If `index.routing.allocation.total_shards_per_node = 4`, then the maximum number of shards each index can allocate per node is four. Therefore, two nodes can only allocate a maximum of eight shards, leaving two shards without available nodes. In production environments, to avoid this issue, the configuration item `index.routing.allocation.total_shards_per_node` is set to a negative number, thus not limiting the maximum number of shards each index can allocate per node.

[0138] After setting the maximum number of shards that each index can allocate to each node without limitation, you can then configure the number of shards for each round. Alternatively, after setting the number of shards for each round, you can set the configuration item `index.routing.allocation.total_shards_per_node` to a negative number to ensure that the number of shards that the scaling node can accommodate is sufficient.

[0139] In a real production environment, the second target parameter of the index template also needs to be configured to be empty, that is, the nodes corresponding to the data shards of the index built based on the index template are not prohibited from being allocated. In other words, the node identifier corresponding to the expansion node, which is configured as the second target parameter when creating a new global matching index template, will be invalidated, and the data shards corresponding to the index will not be restricted from being allocated to the expansion node.

[0140] The total number of fragments that the expansion node can accommodate is M; the number of weight fragments allocated in each round is reduced by K compared to the previous round, and the number of weight fragments allocated in the first round is MK.

[0141] Initially, during the first round of weight allocation, there is no previous round. Therefore, the total number of weights that the expanded node can accommodate can be recorded as the number of weights in the previous round. In other words, the number of weights allocated in the first round is MK.

[0142] As shown in the example above, the total number of weight pieces M is 15. The number of weight pieces allocated in each round is reduced by 2. Therefore, the number of weight pieces set when initially establishing the weight index is 15-2, which is 13, meaning the first round of weight piece allocation is 13. When establishing the weight index for the second time, the number of weight pieces is set to 13-2, which is 11, meaning the second round of weight piece allocation is 11. The third round of weight piece allocation can be set to 11-2, which is 9, meaning the third round of weight piece allocation is 9… and so on. When the number of weight pieces is reduced to 2 or 1, that is, less than or equal to the set reduction number, the weight piece allocation ends, completing the round-by-round allocation of weight pieces for the entire cluster.

[0143] If, as shown in the example above, the calculated total number of shards M that the expanded node can accommodate is 15, and the number of shards reduced in each round of allocation is set to 4, then the number of shards set when initially establishing the weight index is 15 - 4, which is 11, meaning the number of shards allocated in the first round is 11. When establishing the weight index for the second time, the number of shards set is 11 - 4, which is 7, meaning the number of shards allocated in the second round is 7. The number of shards for the third round can be set to 7 - 3, which is 3, meaning the number of shards allocated in the third round is 11. At this point, the number of shards is 3, which is less than the reduction of 4 in each round, ending the weight shard allocation and completing the round-by-round allocation of weight shards for the entire cluster.

[0144] In other words, if the total number of shards that the expanded node can accommodate is M, and the number of shards allocated in each round is reduced by K, then the number of shards set when the weight index is first established is MK, which is the number of shards allocated in the first round; the number of shards set when the weight index is established for the second time is (MK)-K, which is the number of shards allocated in the second round; the number of shards set when the weight index is established for the third time is ((MK)-K)-K, which is the number of shards allocated in the third round... and so on. When the number of shards is reduced to less than or equal to K, the weight shard allocation ends, completing the round-by-round allocation of weight shards for the entire cluster.

[0145] When creating the weight index, the weight segments need to be set up simultaneously. These weight segments do not contain any data; they only serve as placeholders. In practical applications, the number of weight segments, K, reduced in each round can be set according to actual needs.

[0146] S104-3, Distribute the weight pieces to the expansion node.

[0147] This step is used to distribute the weight pieces to the expansion node.

[0148] As shown in the example above, when the weight index is first established, the number of weight fragments allocated is 15-2, which is 13. The 13 weight fragments are distributed to 3 expansion nodes. The number of weight fragments allocated to the 3 expansion nodes can be 5, 5, and 3, or 5, 4, and 4. Then, the subsequent steps of starting the cluster data balancing of the entire cluster are executed.

[0149] When establishing the weight index for the second time, the number of weight fragments allocated is set to 13-2, which is 11. These 11 weight fragments can be distributed among the three expansion nodes in a 5-3-3, 5-4-2, or 5-5-1 configuration. There are many ways to do this, not limited to any one of them; then, the cluster data balancing process for the entire cluster is restarted.

[0150] This process continues until the number of weight pieces is reduced to 2 or 1, at which point the weight pieces are distributed to the expansion node.

[0151] In other words, when the weight index is first established, the number of weight fragments allocated is MK, and the MK weight fragments are distributed to Y_nodes expansion nodes; then, the subsequent steps of starting the cluster data balancing of the entire cluster are executed.

[0152] When the weight index is established for the second time, the number of weight fragments allocated is (MK)-K. These (MK)-K weight fragments are then distributed to the Y_nodes expansion nodes. Finally, the cluster data balancing for the entire cluster is restarted.

[0153] This process continues until the number of weight pieces in each round is reduced to less than or equal to K, at which point the weight pieces are distributed to the expansion node.

[0154] It's important to note that the value of K may not be the same in each round. Furthermore, in the final step, since the total number of slices may not be divisible by K, the value of K in the final step is likely to differ from that in previous rounds. When the number of slices in a round decreases to less than or equal to K, the round ends, and node allocation is completed.

[0155] This concludes the explanation of the process of allocating weight fragments to the expansion node. After allocating weight fragments to the expansion node in each round, data fragments containing actual data need to be allocated to the expansion node.

[0156] S104-4 After completing the weight shard allocation, perform cluster data balancing and allocate data shards with real data to the expansion nodes.

[0157] This step, following the addition of weight sharding in step S104-3 above, initiates cluster data balancing for the entire cluster. After enabling cluster data balancing, data shards containing real data are allocated to the expansion nodes; that is, real data is allocated to the newly added expansion nodes. The real data includes some newly added data and some data from the original cluster. Specifically, some newly added data is written to the expansion nodes, while the data corresponding to the indexes on some original nodes is migrated to the expansion nodes.

[0158] After the cluster completes the weight shard allocation in step S104-3, the configuration item `cluster.routing.rebalance.enable` is set to `all`, allowing shard balancing for all types of shards, thus enabling the cluster data balancing function. At this time, data from the data shard corresponding to the new index is written to the K empty slots. Simultaneously, some data from the data shards corresponding to the original indexes in the cluster may also be migrated to these K empty slots. The cluster automatically starts the data balancing process, gradually writing / migrating data shards with real data to these K empty slots, waiting for the cluster data balancing to complete, thus completing this round of cluster data balancing.

[0159] After completing one round of cluster data balancing—that is, after each round, or before starting the next round—cluster data balancing needs to be disabled. This prevents the cluster from automatically balancing data after creating new indexes and allocating shards, which could misallocate data shards containing actual data to the shards. Additionally, except for the first round, after disabling cluster data balancing, the shard indexes created in the previous round need to be deleted, and the shards set based on those indexes will also be deleted, leaving only the data shards containing the actual data being written on the scaling nodes.

[0160] When the number of weight pieces in a round decreases to less than or equal to K, the round ends, and the node allocation is completed.

[0161] In a real production environment, once the node expansion is complete, the globally matching index template created at the beginning is deleted, thus ending the entire expansion process.

[0162] The above is a detailed description of the node expansion method for a distributed search engine provided in the first embodiment of this application. By pre-allocating nodes in rounds, the weights are distributed to the expansion nodes in rounds to balance the cluster data. Data with real data is distributed to the expansion nodes in rounds to realize in-situ expansion and on-demand expansion of the cluster, avoiding resource waste.

[0163] Reference Figure 4 The diagram illustrates an example of node expansion for a distributed search engine provided in the first preferred embodiment of this application.

[0164] As mentioned in the previous example, in Figure 4-1, the current cluster is in a balanced state, with 2 nodes and 10 data shards. The average number of data shards per node is 5. The solid black ellipse represents a data shard in the cluster during the balanced state, containing actual data. The figure shows that the current cluster's data shards can no longer hold data and urgently need expansion. In Figure 4-2, due to performance limitations, the current cluster can no longer write data, requiring node expansion. In this example, 3 nodes are being expanded. Based on previous calculations, the total number of shards used for expansion is 15. As explained earlier, if cluster data balancing is performed directly, all data from the new index's corresponding data shard will be written to the new node, and data shards from the original cluster nodes will migrate to the new node, causing the new node to overheat and impacting cluster performance. Therefore, data needs to be written in rounds. At this point, data shards with empty values ​​are set, i.e., the weight shards, which contain no data. In the first round of adding weight shards to the expansion nodes, a weight index is set, and the number of weight shards at that index is set to 13. These 13 weight shards are distributed among the 3 expansion nodes, leaving 2 empty slots on the newly added expansion nodes. At this point, cluster data balancing is initiated. As shown in Figure 4-3, according to the data balancing principle, these two empty shards begin receiving data. On the one hand, they receive some data from the data shards on previous nodes that may have migrated to these two shards; on the other hand, they receive some data from newly generated data that may be written to these two newly added shards, until the first round of cluster data balancing is completed. Since the weight shards are already occupied, no other data will be written to them. Thus, in this round, data is only written to the two empty slots.

[0165] After the cluster data balancing is complete, the first round of data balancing ends. Before starting the second round of data balancing, the cluster data balancing is shut down, and the weight indexes created at the beginning of the first round and the weight shards allocated based on the weight indexes are deleted to prepare for the second round of data balancing.

[0166] In Figure 4-4, the second round of data balancing is initiated. Since two empty slots were already filled with real data in the first round, the number of weight shards is set to 11 based on the new weight index in the second round. These 11 placeholder weight shards are distributed to the remaining 13 empty slots on the three expansion nodes. After distribution, the newly added expansion nodes have two empty slots, at which point cluster data balancing is initiated. In Figure 4-5, as mentioned earlier, some data from the previous nodes will be migrated to these two empty slots, and some newly generated data will be written to these two empty slots. Therefore, this round only writes data to the two empty slots.

[0167] After the second round of cluster balancing is completed, the cluster data balancing is shut down, and the weight indexes created in the second round and the weight shards allocated based on the weight indexes are deleted to prepare for the third round of data balancing.

[0168] Following this pattern, when only one empty slot remains in the last round, the cluster data is balanced and the weight index is deleted. At this point, all cluster data has been balanced, completing the round-by-round allocation of the entire cluster data.

[0169] The method provided in the first embodiment of this application, in a distributed search engine, adds expansion nodes to the cluster, distributes empty data shards to the expansion nodes in rounds to achieve cluster data balancing, and distributes data shards with real data to the expansion nodes in rounds, realizing in-situ expansion and on-demand expansion of the cluster. Furthermore, this method avoids writing data to expansion nodes within a concentrated time period, thus preventing impact on data write performance.

[0170] Corresponding to the method for expanding the node capacity of a distributed search engine provided in the first embodiment of this application, the second embodiment of this application provides an apparatus for expanding the node capacity of a distributed search engine.

[0171] Reference Figure 5 This document illustrates a flowchart of an apparatus for scaling up nodes in a distributed search engine, according to a second embodiment of this application. Since the apparatus embodiment is substantially similar to the method embodiment, it is described simply; relevant parts can be found in the corresponding descriptions of the method embodiment.

[0172] Figure 5 The device shown is a node scaling device for a distributed search engine, comprising:

[0173] The information acquisition unit 501 acquires the current information of the cluster of the distributed search engine and the information of the expansion nodes;

[0174] The sharding calculation unit 502 calculates the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node.

[0175] The node addition unit 503 adds the expansion node to the cluster;

[0176] The data sharding allocation unit 504 performs multiple rounds of node pre-allocation. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight shards, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight shards are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight shards corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round; when N reaches a preset value, the node allocation is completed.

[0177] Optionally, before adding the expansion node to the cluster, the method further includes: disabling the data balancing function of the cluster.

[0178] Optionally, before performing multiple rounds of node pre-allocation, the method further includes: creating a globally matching index template and configuring the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the nodes that cannot be allocated to the data shards corresponding to the index established based on the index template.

[0179] Optionally, after establishing the weight index in each round and before performing data balancing on the cluster, the method further includes:

[0180] Configure the second target parameter of the index template to be empty.

[0181] Optionally, after the node allocation is completed, the method further includes: deleting the globally matching index template.

[0182] Optionally, the current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes.

[0183] The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes:

[0184] Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state.

[0185] Based on the average number of data shards and the number of expansion nodes, the total number M of shards that the expansion nodes can accommodate is calculated.

[0186] Optionally, the method further includes: in each round of node pre-allocation steps, before the step of establishing the weight index, disabling the data balancing function of the cluster.

[0187] Optionally, the method further includes: in each round of node pre-allocation step, before the step of establishing the weight index, deleting the weight index established in the previous round of node pre-allocation.

[0188] Optionally, the node pre-allocation in each round further includes:

[0189] The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

[0190] The third embodiment of this application provides an electronic device, such as... Figure 6 As shown, this is a schematic diagram of the logical structure of an electronic device provided in the third embodiment of this application. The electronic device includes:

[0191] Processor 601;

[0192] After storing the program for the node expansion method of the distributed search engine in memory 602, the following steps are executed:

[0193] Obtain the current information of the distributed search engine cluster and the information of the expansion nodes;

[0194] Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated.

[0195] Add the expansion node to the cluster;

[0196] Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index; and performing data balancing on the cluster.

[0197] Wherein, N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round;

[0198] When N reaches the preset value, the node allocation is completed.

[0199] Optionally, before adding the expansion node to the cluster, the method further includes: disabling the data balancing function of the cluster.

[0200] Optionally, before performing multiple rounds of node pre-allocation, the method further includes: creating a globally matching index template and configuring the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the nodes that cannot be allocated to the data shards corresponding to the index established based on the index template.

[0201] Optionally, after establishing the weight index in each round and before performing data balancing on the cluster, the method further includes:

[0202] Configure the second target parameter of the index template to be empty.

[0203] Optionally, after the node allocation is completed, the method further includes: deleting the globally matching index template.

[0204] Optionally, the current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes.

[0205] The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes:

[0206] Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state.

[0207] Based on the average number of data shards and the number of expansion nodes, the total number M of shards that the expansion nodes can accommodate is calculated.

[0208] Optionally, the method further includes: in each round of node pre-allocation steps, before the step of establishing the weight index, disabling the data balancing function of the cluster.

[0209] Optionally, the method further includes: in each round of node pre-allocation step, before the step of establishing the weight index, deleting the weight index established in the previous round of node pre-allocation.

[0210] Optionally, the node pre-allocation in each round further includes:

[0211] The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

[0212] The fourth embodiment of this application provides a non-volatile computer-readable storage medium that stores a program for a node expansion method of a distributed search engine. This program is executed by a processor to perform the following steps:

[0213] Obtain the current information of the distributed search engine cluster and the information of the expansion nodes;

[0214] Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated.

[0215] Add the expansion node to the cluster;

[0216] Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node, wherein the first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragments corresponding to the weight index; and performing data balancing on the cluster.

[0217] Wherein, N is a positive integer less than M, and the N corresponding to the node pre-allocation in each round is smaller than the N corresponding to the node pre-allocation in the previous round;

[0218] When N reaches the preset value, the node allocation is completed.

[0219] Optionally, before adding the expansion node to the cluster, the method further includes: disabling the data balancing function of the cluster.

[0220] Optionally, before performing multiple rounds of node pre-allocation, the method further includes: creating a globally matching index template and configuring the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the nodes that cannot be allocated to the data shards corresponding to the index established based on the index template.

[0221] Optionally, after establishing the weight index in each round and before performing data balancing on the cluster, the method further includes:

[0222] Configure the second target parameter of the index template to be empty.

[0223] Optionally, after the node allocation is completed, the method further includes: deleting the globally matching index template.

[0224] Optionally, the current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes.

[0225] The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes:

[0226] Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state.

[0227] Based on the average number of data shards and the number of expansion nodes, the total number M of shards that the expansion nodes can accommodate is calculated.

[0228] Optionally, the method further includes: in each round of node pre-allocation steps, before the step of establishing the weight index, disabling the data balancing function of the cluster.

[0229] Optionally, the method further includes: in each round of node pre-allocation step, before the step of establishing the weight index, deleting the weight index established in the previous round of node pre-allocation.

[0230] Optionally, the node pre-allocation in each round further includes:

[0231] The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

[0232] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0233] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0234] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0235] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0236] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for scaling up nodes in a distributed search engine, characterized in that, include: Obtain the current information of the distributed search engine cluster and the information of the expansion nodes; Based on the current information of the cluster and the information of the expansion node, the total number M of shards that the expansion node can accommodate is calculated. Disable the data balancing function of the cluster; Add the expansion node to the cluster; Multiple rounds of node pre-allocation are performed. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight fragments, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight fragments are allocated to the expansion node. Here, the weight index refers to the index that occupies storage space in the expansion node, and the weight fragment corresponding to the weight index is a data fragment with empty data. The first target parameter is used to characterize the nodes that are allowed to be allocated the weight fragment corresponding to the weight index. Data balancing is performed on the cluster. Here, N is a positive integer less than M, and the N corresponding to each round of node pre-allocation is smaller than the N corresponding to the previous round of node pre-allocation. When N reaches the preset value, the node allocation is completed.

2. The method according to claim 1, characterized in that, Before performing multiple rounds of node pre-allocation, the method further includes: Create a new globally matching index template, and configure the second target parameter of the index template as the node identifier corresponding to the expansion node; wherein, the second target parameter is used to characterize the node that cannot be allocated to the data shard corresponding to the index built according to the index template.

3. The method according to claim 2, characterized in that, After each round of establishing the weight index and before performing data balancing on the cluster, the method further includes: Configure the second target parameter of the index template to be empty.

4. The method according to claim 3, characterized in that, After completing the node allocation, the method further includes: Delete the globally matching index template.

5. The method according to claim 1, characterized in that, The current information of the cluster includes the current number of nodes and the current number of data shards; the information of the expansion node includes the number of expansion nodes. The step of calculating the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node includes: Based on the current number of nodes and the current number of data shards in the cluster, the average number of data shards per node is calculated when the distributed search engine is in a balanced state. Based on the average number of data shards and the number of expansion nodes, the total number M of shards to be accommodated by the expansion nodes is calculated.

6. The method according to claim 1, characterized in that, The method further includes: In each round of node pre-allocation steps, before the step of establishing the weight index, the data balancing function of the cluster is turned off.

7. The method according to claim 1, characterized in that, The method further includes: In each round of node pre-allocation steps, before the step of establishing the weight index, the weight index established in the previous round of node pre-allocation is deleted.

8. The method according to claim 1, characterized in that, The node pre-allocation in each round also includes: The third objective parameter of the weight index is configured to be unlimited, and the third objective parameter configuration is used to characterize the maximum number of shards that each index can be allocated to on each node.

9. A device for scaling up nodes in a distributed search engine, characterized in that, include: The information acquisition unit acquires the current information of the distributed search engine cluster and the information of the expansion nodes; The sharding calculation unit calculates the total number M of shards that the expansion node can accommodate based on the current information of the cluster and the information of the expansion node. The node addition unit disables the data balancing function of the cluster and adds the expansion node to the cluster; The data sharding allocation unit performs multiple rounds of node pre-allocation. Each round of node pre-allocation includes: establishing a weight index, dividing the weight index into N weight shards, and configuring the first target parameter of the weight index as the node identifier corresponding to the expansion node, so that the N weight shards are allocated to the expansion node. Here, the weight index refers to the index that occupies storage space in the expansion node, and the weight shard corresponding to the weight index is a data shard with empty data. The first target parameter is used to characterize the nodes that are allowed to be allocated the weight shard corresponding to the weight index; performing data balancing on the cluster; wherein N is a positive integer less than M, and the N corresponding to each round of node pre-allocation is smaller than the N corresponding to the previous round of node pre-allocation; when N reaches a preset value, the node allocation is completed.

10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements a method for node expansion of a distributed search engine as described in any one of claims 1-8.

11. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for node expansion of a distributed search engine as described in any one of claims 1-8.

Images