Request processing method and distributed search engine system

Abstract

The invention provides a request processing method and a distributed search engine system, the method is used for a main node in the distributed search engine system, the distributed search engine system further comprises a coordination node and a plurality of index groups, and each index group comprises a plurality of data nodes. The method comprises the following steps: receiving a target batch writing request which is from a coordination node and comprises a target index and a plurality of pieces of document data; selecting a target index group from a plurality of index groups according to the target index; selecting M data nodes from a plurality of data nodes included in the target index group; selecting one data node from the M data nodes according to the load information of the M data nodes to obtain a first data node; the node information of the first data node is sent to the coordination node, and the node information is used for the coordination node to send the target batch writing request to the first data node, so that the multiple pieces of document data are written into the first data node. By adopting the method, the load of the cluster is reduced while the load balance is ensured.

Description

Technical Field

[0001] This application relates to the field of search engine technology, and in particular to a request processing method and a distributed search engine system. Background Technology

[0002] In a distributed search engine system, the client provides a batch write interface. To reduce the number of network requests, multiple documents from the same index can be sent to the cluster via a single network request through this interface. Upon receiving this network request, the coordinating node in the cluster distributes the documents one by one evenly across the data nodes corresponding to the index to achieve load balancing among different data nodes. Since each document requires a separate write request, the number of requests within the cluster is large, resulting in a significant amount of input / output (I / O) and consequently, a high cluster load. Summary of the Invention

[0003] This application provides a request processing method and a distributed search engine system that can reduce the load on the cluster while ensuring load balancing.

[0004] In a first aspect, this application provides a request processing method applied to a master node in a distributed search engine system, wherein the distributed search engine system further includes a coordinating node and multiple index groups, each of the multiple index groups including multiple data nodes, and the method includes:

[0005] Receive a target batch write request from the coordinating node, the target batch write request including a target index and multiple document data;

[0006] Select a target index group from the plurality of index groups based on the target index;

[0007] Select M data nodes from the multiple data nodes included in the target index group, where M is an integer greater than 1;

[0008] Based on the load information of the M data nodes, one data node is selected from the M data nodes to obtain the first data node;

[0009] The node information of the first data node is sent to the coordinating node. The node information is used by the coordinating node to send the target batch write request to the first data node so as to write the multiple document data to the first data node.

[0010] Secondly, this application also provides a request processing apparatus applied to a master node in a distributed search engine system, the distributed search engine system further including a coordinating node and multiple index groups, each of the multiple index groups including multiple data nodes, the apparatus comprising:

[0011] The receiving module is configured to receive a target batch write request from the coordinating node, the target batch write request including a target index and multiple document data;

[0012] The first selection module is used to select a target index group from the plurality of index groups according to the target index;

[0013] The second selection module is used to select M data nodes from the multiple data nodes included in the target index group, where M is an integer greater than 1;

[0014] The third selection module is used to select one data node from the M data nodes based on the load information of the M data nodes to obtain the first data node;

[0015] The sending module is used to send node information of the first data node to the coordinating node. The node information is used by the coordinating node to send the target batch write request to the first data node so as to write the multiple document data to the first data node.

[0016] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0017] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0018] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0019] Sixthly, this application also provides a distributed search engine system, including a client, a master node, a coordinating node, and multiple index groups, wherein each index group includes multiple data nodes;

[0020] The client is used to send a target batch write request to the coordinating node, the target batch write request including a target index and multiple document data;

[0021] The coordinating node is used to send the target batch write request to the master node;

[0022] The master node is configured to select a target index group from the plurality of index groups based on the target index, select M data nodes from the plurality of data nodes included in the target index group, select one data node from the M data nodes based on the load information of the M data nodes to obtain a first data node, and send the node information of the first data node to the coordinating node, where M is an integer greater than 1.

[0023] The coordinating node is also configured to send the target batch write request to the first data node based on the node information;

[0024] The first data node is used to write the multiple document data into the main shard.

[0025] In this application, the master node receives a target batch write request from the coordinating node, which includes a target index and multiple document data. Based on the target index, it selects a target index group from multiple index groups, then selects M data nodes from the multiple data nodes included in the target index group. Based on the load information of the M data nodes, it selects one data node from the M data nodes to obtain a first data node, and sends the node information of the first data node to the coordinating node. This first data node's node information is used by the coordinating node to send the target batch write request to the first data node, so that multiple document data can be written to the first data node. As can be seen, after receiving the batch write request, the coordinating node sends the batch write request to the master node, which determines the data node to be written to. Then, it sends the node information of the determined data node to the coordinating node, which then sends the batch write request to the determined data node based on the node information. Since the cluster can send all document data in the batch write request to the data node through only three intra-cluster communications after receiving the batch write request, the amount of I / O within the cluster can be reduced, thereby reducing the cluster load. In addition, when determining data nodes, the master node can avoid cluster write performance fluctuations based on the load information of the data nodes, and can ensure load balancing among different data nodes. Therefore, it can reduce the cluster load while ensuring load balancing and improve the stability of the cluster. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0028] Figure 2 This is a flowchart illustrating a request processing method provided in an embodiment of this application;

[0029] Figure 3 This is an interactive schematic diagram of a request processing method provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of a master node provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of a request processing device provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0035] This application provides a request processing method and a distributed search engine system that can reduce the load on the cluster while ensuring load balancing.

[0036] To better understand the embodiments of this application, the network architecture of this application will be described below.

[0037] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture may include multiple clients 101, coordinating nodes 102, multiple master nodes 103, and multiple data nodes 104. Clients 101 and coordinating nodes 102, coordinating nodes 102 and master nodes 103, coordinating nodes 102 and data nodes 104, and master nodes 103 and data nodes 104 can be connected via a network. This network can be a local area network (LAN), a wide area network (WAN), or a combination of both.

[0038] The network architecture may include multiple index groups 11. Each index group 11 includes multiple data nodes 104. Multiple data nodes 104 belonging to the same index group 11 store data of different documents indexed by the same index.

[0039] The coordinating node 102, the master node 103, and multiple data nodes 104 form a cluster, i.e., a node cluster.

[0040] Among multiple master nodes 103, one master node 103 serves as the primary master node 103, while the other master nodes 103 serve as backup master nodes 103. In the event of a failure of the primary master node 103, the backup master node 103 can be used as the primary master node 103, thus ensuring the availability of the cluster.

[0041] Client 101 can generate a batch write request based on data from multiple documents with the same index, and then send this batch write request to coordinating node 102. This batch write request can include the data and index of these multiple documents. The number of documents included in the batch write request can be hundreds, thousands, or even tens of thousands. Furthermore, the batch write request can also carry a request type, which indicates whether the write request is a batch write request or a normal write request. A batch write request is a write request that includes a number of documents greater than or equal to a certain threshold. A normal write request is a write request that includes a number of documents less than the threshold. A batch write request includes a number of documents significantly greater than the number of documents included in a normal write request.

[0042] After receiving a batch write request, the coordinating node 102 can send a batch write request to the master node 103.

[0043] In some embodiments, when a write request carries a request type, the coordinating node 102 can first obtain the request type of the write request. If the request type indicates that the write request is a batch write request, it can send the write request to the master node 103. If the request type indicates that the write request is a non-batch write request, it can calculate the hash value of each document included in the write request based on the identifier of each document included in the write request, and send the data of each document to the corresponding data node in the corresponding index value based on the hash value of each document.

[0044] In some embodiments, after receiving a write request, the coordinating node 102 can determine the processing mechanism of the current write request. If the processing mechanism of the current write request is a batch request processing mechanism, it can send the write request to the master node 103. If the processing mechanism of the current write request is a non-batch request processing mechanism, it can calculate the hash value of each document included in the write request based on the identifier of each document included in the write request, and send the data of each document to the corresponding data node in the corresponding index value based on the hash value of each document.

[0045] After receiving a batch write request, the master node 103 can first determine the index group corresponding to the index included in the batch write request. Then, it can select M data nodes 104 from the multiple data nodes 104 included in the index group. Furthermore, based on the load information of the M data nodes 104, it can select one data node 104 from the M data nodes 104 and send the node information of this data node 104 to the coordinating node 102. M is a fixed value that can be set by the user or preset.

[0046] After receiving the node information of the data node 104, the coordinating node 102 can send a batch write request to the data node 104 corresponding to the node information in the index group included in the batch write request.

[0047] Data node 104 is deployed with multiple primary shards. After receiving a batch write request, data node 104 can calculate the hash value of each document included in the batch write request based on the identifier of each document included in the batch write request. Based on the hash value of each document, it can write the data of each document to the corresponding primary shard. After the data of all documents has been written to the corresponding primary shards, it can send an indication message to coordinating node 102 to indicate that the batch write request has been completed.

[0048] After receiving the instruction information, the coordinating node 102 can send the instruction information to the client 101.

[0049] After receiving the indication information, client 101 can determine that the batch write request has been processed, that is, the batch write request has been completed.

[0050] The aforementioned nodes can be terminal devices or servers. Terminal devices or servers can perform corresponding processing by running computer programs. For example, a computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded into any APP, where the user can control whether to run or close it. In short, the aforementioned computer program can be any form of application, module, or plugin.

[0051] The server can be a standalone physical server, a server cluster consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, big data, and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart TV, smartwatch, etc., but is not limited to these. The terminal device and the server can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0052] It should be understood that Figure 1 This is an exemplary illustration of the network architecture of this application and does not limit the network architecture of this application.

[0053] Figure 2 This is a flowchart illustrating a request processing method provided in an embodiment of this application. The request processing method is applied to the master node of a distributed search engine system. The distributed search engine system also includes a coordinating node and multiple index groups, each of which includes multiple data nodes. Figure 2 As shown, the request processing method may include steps 201-205.

[0054] 201. Receive a target batch write request from the coordinating node, which includes the target index and multiple document data.

[0055] A target batch write request is any batch write request sent by the coordinating node to the master node. A target batch write request can include a target index and multiple document data. The target index is the index of the multiple document data. The document data is the data of the documents. The multiple document data consists of the data of multiple documents. The multiple documents share the same index, which is the target index.

[0056] A batch write request can be a Bulk write request or other batch write requests with equivalent functionality.

[0057] 202. Select the target index group from multiple index groups based on the target index.

[0058] After receiving the target batch write request, the target batch write request can be parsed to obtain the target index. The target index group can then be selected from multiple index groups based on the target index.

[0059] 203. Select M data nodes from the multiple data nodes included in the target index group.

[0060] The master node can select M data nodes from the multiple data nodes included in the target index group, that is, select M data nodes from all data nodes included in the target index group, so as to narrow down the selection range of data nodes and thus improve the efficiency of data node determination.

[0061] The M data nodes can be the M data nodes that store the fewest documents among the multiple data nodes included in the target index group, or the M data nodes among the multiple data nodes included in the target index group that have no pending batch write requests, or the M data nodes among the multiple data nodes included in the target index group that are fault-free.

[0062] M is an integer greater than 1. For example, M can be 10, 9, 15, or any other value.

[0063] 204. Based on the load information of M data nodes, select one data node from the M data nodes to obtain the first data node.

[0064] Data node load information refers to information related to the load on a data node, indicating the level of resource consumption undertaken by the data node. Data node load information can include Central Processing Unit (CPU) utilization. It can also include memory utilization. CPU utilization is the percentage of time the CPU is processing tasks per unit of time. Memory utilization is the percentage of total memory that is currently in use.

[0065] To ensure load balancing among different data nodes, one data node can be selected from the M data nodes based on their load information to obtain the first data node.

[0066] 205. Send the node information of the first data node to the coordinating node. The node information of the first data node is used by the coordinating node to send a target batch write request to the first data node so that multiple document data can be written to the first data node.

[0067] The node information of the first data node can be the identifier of the first data node, or it can be other information that uniquely identifies the first data node.

[0068] After obtaining the first data node, the node information of the first data node can be sent to the coordinating node, so that the coordinating node can send a target batch write request to the first data node based on the node information, so that multiple document data can be written to the first data node.

[0069] exist Figure 2 In the request processing method shown, after receiving a batch write request, the coordinating node sends the batch write request to the master node. The master node determines the data nodes to be written to in the batch write request and then sends the determined data node information to the coordinating node. The coordinating node then sends the batch write request to the determined data nodes based on the node information. Since the cluster can send all document data in the batch write request to the data nodes through only three intra-cluster communications after receiving the batch write request, the amount of I / O within the cluster can be reduced, thereby reducing the cluster load. In addition, when determining the data nodes, the master node can avoid cluster write performance jitter based on the load information of the data nodes, ensuring load balancing among different data nodes. Therefore, while ensuring load balancing, the cluster load can be reduced, and the cluster stability can be improved.

[0070] In some embodiments, selecting M data nodes from the plurality of data nodes included in the target index group includes: if there are data nodes in the plurality of data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, removing the data nodes in the plurality of data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, to obtain N data nodes, where N is an integer greater than 1 and less than M; selecting M data nodes from the N data nodes.

[0071] After selecting the target index group, it can be determined whether any data nodes in the target index group have corresponding incomplete batch write requests. If no data nodes in the target index group have corresponding incomplete batch write requests, it means that all data nodes in the target index group can participate in the selection, and M data nodes can be selected from the multiple data nodes included in the target index group. If data nodes in the target index group have corresponding incomplete batch write requests, in order to avoid the extreme case of multiple batch write requests writing to the same data node simultaneously, the data node will not participate in the selection of data nodes corresponding to batch write requests until the incomplete batch write request corresponding to a data node is completed. Therefore, we can first determine the data nodes in the multiple data nodes included in the target index group that have corresponding incomplete batch write requests, obtaining at least one data node. Then, we can remove at least one data node from the multiple data nodes included in the target index group, obtaining N data nodes, and then select M data nodes from the N data nodes. N is an integer greater than 1 and less than M.

[0072] An incomplete batch write request can be understood as the client that sent the batch write request not receiving the information that the batch write request has been completed.

[0073] In the above embodiments, if there are data nodes in the target index group that correspond to incomplete batch write requests, these data nodes are removed, resulting in N data nodes. M data nodes are then selected from these N data nodes. By first removing the data nodes with incomplete batch write requests from the index group, the extreme situation of multiple batch write requests simultaneously writing to the same data node can be avoided, thus ensuring load balancing.

[0074] In some embodiments, selecting M data nodes from a plurality of data nodes included in the target index group includes: obtaining the number of documents stored in each data node among the plurality of data nodes included in the target index group, and selecting the M data nodes with the fewest stored documents from the plurality of data nodes included in the target index group.

[0075] The more documents a data node stores, the more data it can potentially store, and the more memory it might consume. Therefore, we can first obtain the number of documents stored in each data node of the target index group, i.e., the total number of documents. Then, we can select the M data nodes with the fewest stored documents from the multiple data nodes of the target index group.

[0076] In one scenario, the data nodes in the target index group can be sorted in descending order according to the number of documents stored, resulting in a descending list. The last M data nodes in the descending list can then be selected.

[0077] In another scenario, the data nodes in the target index group can be sorted in ascending order according to the number of documents stored, resulting in an ascending list. The first M data nodes can then be selected from the ascending list.

[0078] In the above embodiments, the number of documents stored in each data node among the multiple data nodes included in the target index group is obtained, and the M data nodes with the fewest stored documents are selected from the multiple data nodes included in the target index group. Since the M data nodes with the fewest stored documents can be selected in a short time based on the number of documents stored in the multiple data nodes included in the target index group, the efficiency of determining data nodes can be improved.

[0079] In some embodiments, selecting M data nodes from the multiple data nodes included in the target index group includes: if there are data nodes in the data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, removing the data nodes in the multiple data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, obtaining N data nodes, obtaining the number of documents stored in each of the N data nodes, and selecting the M data nodes with the fewest number of documents stored from the N data nodes.

[0080] After selecting the target index group, it can be determined whether there are any data nodes in the target index group that have incomplete batch write requests. If there are no data nodes in the target index group that have incomplete batch write requests, it means that all data nodes in the target index group can participate in the selection. The number of documents stored in each data node in the target index group can be obtained. Then, the M data nodes with the fewest stored documents can be selected from the multiple data nodes in the target index group.

[0081] In cases where there are data nodes with incomplete batch write requests among the data nodes included in the target index group, to avoid the extreme situation where multiple batch write requests simultaneously write to the same data node due to the same selected data node, this data node will no longer participate in the selection of data nodes corresponding to batch write requests until the incomplete batch write request corresponding to a data node is completed. Therefore, we can first determine the data nodes with incomplete batch write requests among the multiple data nodes included in the target index group to obtain at least one data node. Then, we can remove at least one data node from the multiple data nodes included in the target index group to obtain N data nodes. We can obtain the number of documents stored in each of the N data nodes, and select the M data nodes with the fewest stored documents from the N data nodes.

[0082] In the above embodiments, by first removing data nodes in the index group that correspond to incomplete batch write requests, the extreme situation of multiple batch write requests simultaneously writing to the same data node due to the selection of the same data node can be avoided, thus ensuring load balancing. Furthermore, since the M data nodes with the fewest stored documents can be selected in a short time based on the number of documents stored in the N data nodes, the efficiency of data node determination can be improved.

[0083] In some embodiments, selecting one data node from the M data nodes based on the load information of the M data nodes to obtain a first data node includes: selecting one data node from the M data nodes based on the number of documents, network request latency, and load information corresponding to the M data nodes to obtain a first data node.

[0084] The number of documents corresponding to a data node is the number of documents stored on that data node. The network request latency corresponding to a data node is the total time taken for a write request to be sent from the client to the data node, and for the data node to return a message indicating that the write request is complete to the client. The load information corresponding to a data node is the load information for that data node.

[0085] Based on the number of documents, network request latency, and load information corresponding to the M data nodes, one data node can be selected from the M data nodes to obtain the first data node.

[0086] In the above embodiments, when selecting data nodes, not only the load information corresponding to the data node is considered, but also the number of documents and network request latency corresponding to the data node are also considered. Since the network request latency corresponding to the data node is related to the real-time response of the request, and the number of documents corresponding to the data node is related to the memory space of the data node, selecting data nodes based on the corresponding number of documents, network request latency and load information can not only ensure the timeliness and high availability of batch write requests, but also ensure the balance of the final stored data, and can ensure the high availability, stability and dynamic balance of the cluster.

[0087] In some embodiments, selecting a data node from the M data nodes to obtain a first data node based on the number of documents, network request latency, and load information corresponding to the M data nodes includes: determining the document value of each data node based on the number of documents stored in each of the M data nodes; determining the network latency value of each data node based on the network request latency corresponding to each data node; determining the load value of each data node based on the load information corresponding to each data node; determining the sum of the document value, network latency value, and load value of each data node to obtain the cumulative value of each data node; and selecting the data node with the smallest corresponding cumulative value from the M data nodes to obtain the first data node.

[0088] The document value of each data node can be determined based on the number of documents stored in each of the M data nodes. Document weights can be obtained, and the document value of each data node can be determined based on the number of documents stored in each of the M data nodes and its document weight. Alternatively, the document value of each data node can be determined by the product or ratio of the number of documents stored in each data node to its document weight.

[0089] The network latency of each data node can be determined based on the network request latency. The network latency weight of each data node can be determined based on its network request latency. The network latency value of each data node can be determined by combining its network latency weight and network request latency. Alternatively, the network latency value of each data node can be determined by the product or ratio of its network request latency and network latency weight.

[0090] In one scenario, the network latency weight for each data node can be determined based on the network request latency and a network request latency threshold for each data node. If the network request latency for the third data node is greater than or equal to the network request latency threshold, a larger penalty value can be applied, and the network latency weight for the third data node can be designated as the first weight. If the network request latency for the third data node is less than the network request latency threshold, the magnitude of the network request latency does not have a decisive impact; therefore, the network latency weight for the third data node can be designated as the second weight. The first weight is greater than the second weight. The third data node can be any one of the M data nodes.

[0091] In another scenario, the network request latency range for each data node can be determined based on its network request latency. The weight corresponding to each data node's network request latency range can then be determined based on the correlation between these ranges and their corresponding weights, resulting in a network latency weight for each data node. Since different network request latencies have varying impacts on the timeliness of batch write requests and cluster load, the network request latency can be divided into different ranges, each with a different weight. The network request latency range and weight for each data node are positively correlated.

[0092] The load value of each data node can be determined based on its load information. The load weight of each data node can be determined based on its load information. The load value of each data node can be determined based on its load weight and load information. Alternatively, the load value of each data node can be determined by multiplying or dividing its load information by its load weight.

[0093] In one scenario, the load weight for each data node can be determined based on its load information and load threshold. If the load information for the third data node is greater than or equal to the load threshold, a larger penalty value can be applied, thus determining the load weight for the third data node as the third weight. If the load information for the third data node is less than the load threshold, the magnitude of the load information is not decisive; therefore, the load weight for the third data node can be determined as the fourth weight. The third weight is greater than the fourth weight.

[0094] In another scenario, the load range for each data node can be determined based on its load information. The load weight for each data node within its load range can then be determined based on the correlation between load ranges and load weights. Since different load information has varying impacts on cluster write efficiency and query performance, the load information can be divided into different ranges, each corresponding to a different load weight. The load range and load weight for a data node are positively correlated.

[0095] Load weights are used to measure the impact of data node load on batch write requests. Load information is used to avoid randomly or cyclically selecting data nodes that result in high-load data nodes being selected, thereby affecting overall write efficiency and cluster query performance.

[0096] The sum of the document value, network latency value, and load value of each data node can be determined to obtain the cumulative value of each data node. The data node with the smallest cumulative value can be selected from the M data nodes to obtain the first data node.

[0097] For example, the determination of the first data node can be represented as follows:

[0098] result=min((αX1+βY1+γZ1),…(αXM+βYM+γZM)

[0099] Where α is the document weight, Xi is the number of documents stored in the i-th data node, β is the network latency weight, Yi is the network request latency corresponding to the i-th data node, γ is the load weight, and Zi is the load information corresponding to the i-th data node. αXi is the document value of the i-th data node, βYi is the network latency value of the i-th data node, γZi is the load value of the i-th data node, and αXi + βYi + γZi is the cumulative value of the i-th data node. result is the minimum cumulative value among the M data nodes. When Yi is greater than or equal to T, β can be βmax; when Yi is less than T, β can be βmin, and βmax is greater than βmin. When Zi is greater than or equal to L, γ can be γmax; when Zi is less than L, γ can be γmin, and γmax is greater than γmin. i = 1, 2, ..., M.

[0100] In another embodiment, document weights can be obtained. The network latency weight of each data node can be determined based on the network request latency corresponding to each data node. The load weight of each data node can be determined based on the load information corresponding to each data node. The document weight, network latency weight, and load weight of each data node can be used to perform a weighted sum of the number of documents, network request latency, and load information corresponding to each data node to obtain the weighted sum of each data node. The data node with the smallest weighted sum can be selected from the M data nodes to obtain the first data node.

[0101] In the above embodiments, when selecting data nodes, the data node with the smallest weighted sum of corresponding load information, number of documents, and network request latency can be selected. This can avoid selecting data nodes with high load, large number of stored documents, and large network request latency. This not only ensures the timeliness and high availability of batch write requests, but also ensures the balance of the final stored data, thus guaranteeing the high availability, stability, and dynamic balancing of the cluster.

[0102] In some embodiments, the above request processing method may further include: storing the target batch write request to a write cache queue.

[0103] The master node maintains a write cache queue. The write cache queue is a queue that caches incomplete batch write requests.

[0104] After obtaining the first data node, the target batch write requests can be stored in the write cache queue. The mapping between target batch write requests and the first data node can also be stored to indicate that the target batch write requests are written to the first data node.

[0105] The master node can also monitor the completion status of target batch write requests. Once a target batch write request is completed (i.e., written to the first data node), the first data node sends a completion message to the corresponding client. After the client successfully receives this message, it can remove the target batch write request from the cache queue, reducing unnecessary monitoring. Furthermore, the stored mapping between target batch write requests and the first data node can be deleted, saving storage resources.

[0106] In the above embodiments, the target batch write requests are stored in the write cache queue so as to monitor the completion status of the batch write requests. This allows for the reasonable determination of the data nodes to be written for the batch write requests, and ensures load balancing among the data nodes.

[0107] In some embodiments, the above request processing method may further include: in the event of a failure of the second data node, obtaining an incomplete batch write request corresponding to the second data node from the write cache queue, obtaining at least one batch write request, and sending at least one batch write request to the coordinating node, wherein the at least one batch write request is used by the coordinating node to send a corresponding batch write request to the corresponding client.

[0108] The master node can monitor the operational status of data nodes. In the event of a second data node failure, it can determine if the write cache queue contains any incomplete batch write requests corresponding to that data node. If the write cache queue does not contain such requests, no action is required. If the write cache queue does contain such requests, it indicates that there are still incomplete batch write requests for that data node. The master node can then retrieve these requests from the write cache queue, obtaining at least one batch write request. This request can be sent to the coordinating node, which then forwards the corresponding batch write request to the relevant client. Upon receiving the returned batch write request, the client can resend it to the coordinating node. This resends the batch write request to the coordinating node, allowing for the reallocation of data nodes and the writing of the data, thus ensuring the integrity of the final written data.

[0109] In the above embodiments, in the event of a data node failure, the incomplete batch write requests corresponding to the data node will be returned to the client through the coordinating node, so that the client can resend the batch write requests to the coordinating node for rewriting, thus ensuring the integrity of the written data.

[0110] Figure 3 This is an interactive schematic diagram of a request processing method provided in an embodiment of this application. For example... Figure 3 As shown, the request processing method may include the following steps.

[0111] 301. The client sends a target batch write request to the coordinating node, which includes the target index and multiple document data.

[0112] Accordingly, the coordinating node receives a target batch write request from the client, which includes the target index and multiple document data.

[0113] When a client needs to write documents to the cluster, it can generate a target batch write request based on the data of multiple documents with the same index. Then, it can send the target batch write request to the coordinating node so that multiple documents can be sent to the cluster at once, which can reduce the number of communications and thus the number of network requests, thereby saving communication resources.

[0114] The target batch write request may include the target index and multiple document data. For a detailed description, please refer to the relevant description in step 201, which will not be repeated here.

[0115] 302. The coordinating node sends a target batch write request to the master node.

[0116] Accordingly, the master node receives target batch write requests from the coordinating node.

[0117] After receiving a target batch write request from a client, the coordinating node can send the target batch write request to the master node.

[0118] 303. The master node selects the target index group from multiple index groups based on the target index.

[0119] For a detailed description of step 303, please refer to the description of step 202, which will not be repeated here.

[0120] 304. The master node selects M data nodes from the multiple data nodes included in the target index group.

[0121] For a detailed description of step 304, please refer to the description of step 203, which will not be repeated here.

[0122] 305. The master node selects one data node from the M data nodes based on the load information of the M data nodes, and obtains the first data node.

[0123] For a detailed description of step 305, please refer to the description of step 204, which will not be repeated here.

[0124] 306. The master node sends the node information of the first data node to the coordinating node.

[0125] Accordingly, the coordinating node receives node information from the first data node of the master node.

[0126] For a detailed description of step 306, please refer to the description of step 205, which will not be repeated here.

[0127] 307. The coordinating node sends a target batch write request to the first data node based on the node information of the first data node.

[0128] Accordingly, the first data node receives the target batch write request from the coordinating node.

[0129] After receiving the node information of the first data node, the coordinating node can send a target batch write request to the first data node based on the node information.

[0130] 308. Multiple document data from the first data node are written to the primary shard.

[0131] Multiple primary shards are deployed on the first data node. After receiving a target batch write request, the first data node can parse the request to obtain multiple document data. It can calculate the hash value of each document based on its identifier and write each document data to its corresponding primary shard based on its hash value. After all document data has been written to its corresponding primary shard, the first data node can send an indication message to the coordinating node to indicate that the target batch write request has been completed.

[0132] exist Figure 3 In the requested processing method shown, after receiving a batch write request, the coordinating node sends the batch write request to the master node. The master node determines the data nodes to be written to in the batch write request and then sends the determined data node information to the coordinating node. The coordinating node then sends the batch write request to the determined data node based on the node information. Since the cluster can send all document data in the batch write request to the data node through only three intra-cluster communications after receiving the batch write request, the amount of I / O within the cluster can be reduced, thereby reducing the cluster load. Furthermore, when determining the data node, the master node uses the data node's load information to avoid randomly selecting the write data node, which could lead to load imbalance. This ensures load balancing among different data nodes and guarantees high availability and stability of the cluster. Further...

[0133] Figure 4 This is a schematic diagram of a master node structure provided in an embodiment of this application. The master node is illustrated using an Elastic Search (ES) system as an example of a distributed search engine system. Figure 3 As shown, the master node may include a load balancing module. The load balancing module may include an intelligent allocation unit, a background polling unit, a fault tolerance processing unit, a compatibility adaptation unit, a receiving unit, and a sending unit. The intelligent allocation unit is the core unit for maintaining load balancing of batch write requests in the cluster. It is mainly responsible for determining the data nodes to be written for each batch write request based on the current batch write requests. A detailed description can be found in the description of determining the first data node above, and will not be repeated here.

[0134] The background polling unit is a crucial component for load balancing within the cluster. It periodically retrieves the document count, network status, and load information for all data nodes. Based on this information, it maintains a centralized, unified load information table. The period can be every minute, every two minutes, or other intervals. Network status data includes network request latency. Furthermore, the background polling unit can cache incomplete batch write requests through a write cache queue and store the mapping between incomplete batch write requests and their corresponding data nodes. When a data node successfully completes a batch write request and returns a successful response, it can clear the batch write request and its corresponding mapping from the write cache queue. The background polling unit can also work in conjunction with a fault tolerance unit to handle exceptions in extreme cases.

[0135] The fault tolerance unit is a crucial component for handling anomalies within the cluster. In extreme cases, a data node might fail during the writing of some batch write requests. In this situation, the cluster's master node can temporarily remove the data node from the service cluster. The fault tolerance unit can promptly capture this cluster state change information and return all incomplete batch write requests corresponding to that data node, cached in the background polling unit's cache queue, to the client through the coordinating node. If the client has a built-in retry mechanism, it can resend these batch write requests to the coordinating node, allowing the intelligent allocation unit to reallocate new data nodes for these requests, thus ensuring the integrity of the final written data. Furthermore, due to the cluster's data synchronization mechanism, after a document is written to the primary shard of a data node, the data can be automatically and reliably synchronized to the secondary shard, greatly simplifying the design of inter-cluster write fault tolerance. The fault tolerance unit comprehensively ensures high reliability of write data across multiple dimensions, including the cluster, clients, and nodes within the cluster.

[0136] The compatibility adaptation unit is a crucial component for multi-version adaptation within the cluster. The latest Elasticsearch (ES) version is 8.X, while ES versions 5.X, 6.X, and 7.X are also widely used in various production systems. The core algorithm and implementation of the load balancing module are independent of the ES version, inherently possessing high portability and versatility. However, the return values ​​of some calls to the master node's Application Programming Interface (API) may differ. The compatibility adaptation unit can be activated when the master node starts, starting the load balancing module, obtaining the current ES version, and internally calling the compatibility adaptation unit to select different implementations. This normalizes the differences in return value interfaces between different versions, ensuring consistency in the interaction interfaces between units and between the load balancing module and the coordinating node.

[0137] The receiving unit is a basic module that can communicate with other nodes. It can receive requests sent by the coordinating node and, when the background polling unit is invoked, can be responsible for receiving information sent from other data nodes. The sending unit is also a basic module that can communicate with the coordinating node. It can send information about the data nodes to be written, as well as the communication methods within the cluster, to the coordinating node. When the background polling unit is invoked, it can be responsible for sending requests to collect information from data nodes.

[0138] Leveraging Elasticsearch's plugin loading mechanism, the load balancing module supports cold start and hot-swapping. Default on / off parameters can be specified within the plugin, and code has also been written to support dynamic modification of these parameters. Based on this capability, the batch write request processing mechanism can be enabled or disabled by default during cluster cold start. Furthermore, the rules for supporting load balancing for batch write requests can be set directly by sending commands to the Representational State Transfer (REST) ​​API.

[0139] The load balancing module comprises an intelligent allocation unit, a background polling unit, a fault-tolerant processing unit, a compatibility adaptation unit, a receiving unit, and a sending unit. These multiple units work collaboratively and in close coordination, ensuring low coupling and high availability. Within the master node, each unit performs its specific function while cooperating with each other, providing strong fault tolerance and high availability. Furthermore, the load balancing module's batch write request allocation rules support both cold starts and hot-swapping; when batch write requests are low, the batch write request allocation rules can be disabled. Further, the coordinated consistency processing of the fault-tolerant processing unit and the background polling unit enhances and supplements the fault and recovery mechanisms of the distributed system. While optimizing the write mechanism, it provides a reliable fault recovery mechanism at the data node level and reuses the cluster's built-in high-reliability master-slave sharding synchronization mechanism. Moreover, the compatibility adaptation unit ensures broad compatibility across different Elasticsearch versions in various production systems, enabling a single service to be reused in multiple locations without modification, greatly improving the technology's versatility.

[0140] It should be understood that identical or corresponding information in different embodiments can be referenced together, and content and / or steps in different embodiments can be combined with each other. At least two embodiments in different embodiments can be combined with each other.

[0141] Based on the same concept, this application also provides a request processing apparatus for implementing the request processing method described above. The solution provided by the request processing apparatus is similar to the solution described in the above method; therefore, the specific limitations in the following embodiments of the request processing apparatus can be found in the limitations of the request processing method described above, and will not be repeated here.

[0142] Figure 5 This is a schematic diagram of a request processing device provided in an embodiment of this application. The request processing is applied to the master node of a distributed search engine system. The distributed search engine system also includes a coordinating node and multiple index groups, each of which includes multiple data nodes. The request processing device may include:

[0143] The receiving module 501 is used to receive a target batch write request from the coordinating node. The target batch write request includes a target index and multiple document data.

[0144] The first selection module 502 is used to select a target index group from multiple index groups based on the target index;

[0145] The second selection module 503 is used to select M data nodes from the multiple data nodes included in the target index group, where M is an integer greater than 1.

[0146] The third selection module 504 is used to select one data node from the M data nodes based on the load information of the M data nodes to obtain the first data node;

[0147] The sending module 505 is used to send the node information of the first data node to the coordinating node. The node information of the first data node is used by the coordinating node to send a target batch write request to the first data node so as to write multiple document data to the first data node.

[0148] In some embodiments, the second selection module 503 is specifically used to: when there are data nodes in the target index group that correspond to data nodes with incomplete batch write requests, remove the data nodes in the target index group that correspond to data nodes with incomplete batch write requests to obtain N data nodes, where N is an integer greater than 1 and less than M; and select M data nodes from the N data nodes.

[0149] In some embodiments, the second selection module 503 is specifically used to: obtain the number of documents stored in each data node among the multiple data nodes included in the target index group; and select the M data nodes with the fewest stored documents from the multiple data nodes included in the target index group.

[0150] In some embodiments, the third selection module 504 is specifically used to select one data node from the M data nodes based on the number of documents, network request latency and load information corresponding to the M data nodes, to obtain the first data node.

[0151] In some embodiments, the third selection module 504 selects one data node from the M data nodes based on the number of documents, network request latency, and load information corresponding to the M data nodes to obtain a first data node, including: determining the document value of each data node based on the number of documents stored in each of the M data nodes; determining the network latency value of each data node based on the network request latency corresponding to each data node; determining the load value of each data node based on the load information corresponding to each data node; determining the sum of the document value, network latency value, and load value of each data node to obtain the cumulative value of each data node; and selecting the data node with the smallest corresponding cumulative value from the M data nodes to obtain the first data node.

[0152] In some embodiments, the third selection module 504 determines the network latency value of each data node based on the network request latency corresponding to each data node, including: determining the network latency weight corresponding to each data node based on the network request latency corresponding to each data node; and determining the network latency value of each data node based on the network latency weight and the network request latency corresponding to each data node.

[0153] In some embodiments, the third selection module 504 determines the load value of each data node based on the load information corresponding to each data node, including: determining the load weight corresponding to each data node based on the load information corresponding to each data node; and determining the load value of each data node based on the load weight and load information corresponding to each data node.

[0154] In some embodiments, the request processing apparatus may further include: a storage module for storing target batch write requests to a write cache queue.

[0155] In some embodiments, the request processing apparatus may further include: an acquisition module, configured to acquire, in the event of a failure of the second data node, an incomplete batch write request corresponding to the second data node from the write cache queue, thereby obtaining at least one batch write request; and a sending module 505, further configured to send at least one batch write request to the coordinating node, wherein the at least one batch write request is used by the coordinating node to send a corresponding batch write request to the corresponding client.

[0156] Each module in the aforementioned request processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor within the request processing device in hardware form, or stored in the memory of the request processing device in software form, so that the processor can invoke and execute the operations corresponding to each module.

[0157] In one exemplary embodiment, Figure 6 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of this computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external devices via a network connection. When the computer program is executed by the processor, it implements the aforementioned request processing method.

[0158] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In an exemplary embodiment, this application provides a distributed search engine system, which may include a client, a master node, a coordinating node, and multiple index groups, each index group including multiple data nodes; the client is used to send a target batch write request to the coordinating node, the target batch write request including a target index and multiple document data; the coordinating node is used to send the target batch write request to the master node; the master node is used to select a target index group from the multiple index groups according to the target index, select M data nodes from the multiple data nodes included in the target index group, select one data node from the M data nodes according to the load information of the M data nodes to obtain a first data node, and send the node information of the first data node to the coordinating node, where M is an integer greater than 1; the coordinating node is also used to send a target batch write request to the first data node according to the node information of the first data node; the first data node is used to write multiple document data to the main shard.

[0160] In some embodiments, the master node selects M data nodes from the multiple data nodes included in the target index group, including: if there are data nodes in the multiple data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, the master node removes the data nodes in the multiple data nodes included in the target index group that correspond to data nodes with incomplete batch write requests, and obtains N data nodes, where N is an integer greater than 1 and less than M; the master node selects M data nodes from the N data nodes.

[0161] In some embodiments, the master node selects M data nodes from the plurality of data nodes included in the target index group, including: the master node obtaining the number of documents stored in each data node among the plurality of data nodes included in the target index group; and the master node selecting the M data nodes with the fewest stored documents from the plurality of data nodes included in the target index group.

[0162] In some embodiments, the master node selects one data node from the M data nodes based on the load information of the M data nodes to obtain the first data node, including: the master node selects one data node from the M data nodes based on the number of documents, network request latency and load information corresponding to the M data nodes to obtain the first data node.

[0163] In some embodiments, the master node selects one data node from the M data nodes based on the number of documents, network request latency, and load information corresponding to the M data nodes to obtain a first data node. This includes: determining the document value of each data node based on the number of documents stored in each of the M data nodes; determining the network latency value of each data node based on the network request latency corresponding to each data node; determining the load value of each data node based on the load information corresponding to each data node; determining the sum of the document value, network latency value, and load value of each data node to obtain the cumulative value of each data node; and selecting the data node with the smallest corresponding cumulative value from the M data nodes to obtain the first data node.

[0164] In some embodiments, the master node determines the network latency value of each data node based on the network request latency corresponding to each data node, including: determining the network latency weight corresponding to each data node based on the network request latency corresponding to each data node; and determining the network latency value of each data node based on the network latency weight and the network request latency corresponding to each data node.

[0165] In some embodiments, the master node determines the load value of each data node based on the load information corresponding to each data node, including: determining the load weight of each data node based on the load information corresponding to each data node; and determining the load value of each data node based on the load weight and load information corresponding to each data node.

[0166] In some embodiments, the master node is also configured to store target batch write requests to a write cache queue.

[0167] In some embodiments, the master node is further configured to, in the event of a failure of the second data node, retrieve the incomplete batch write requests corresponding to the second data node from the write cache queue, obtain at least one batch write request, and send at least one batch write request to the coordinating node; the coordinating node is further configured to send the corresponding batch write request to the corresponding client.

[0168] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0170] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0171] It should be noted that all information involved in this application is authorized by the user or fully authorized by all parties, and the collection, use and processing of such information must comply with relevant regulations.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A request processing method characterized by comprising: A master node applied to a distributed search engine system, the distributed search engine system further comprising a coordination node and a plurality of index groups, each index group of the plurality of index groups comprising a plurality of data nodes, the method comprising: receiving a target batch write request from the coordination node, the target batch write request comprising a target index and a plurality of document data; selecting a target index group from the plurality of index groups according to the target index; selecting M data nodes from the plurality of data nodes comprised by the target index group, M being an integer greater than 1; selecting a first data node from the M data nodes according to load information of the M data nodes; sending node information of the first data node to the coordination node, the node information being used by the coordination node to send the target batch write request to the first data node so as to write the plurality of document data to the first data node.

2. The method of claim 1, wherein, The selecting M data nodes from the plurality of data nodes comprised by the target index group comprises: in a case where there is a data node corresponding to an uncompleted batch write request among the data nodes comprised by the target index group, excluding the data node corresponding to the uncompleted batch write request from the plurality of data nodes comprised by the target index group to obtain N data nodes, N being an integer greater than 1 and smaller than M; selecting M data nodes from the N data nodes.

3. The method of claim 1, wherein, The selecting M data nodes from the plurality of data nodes comprised by the target index group comprises: obtaining a number of documents stored by each data node of the plurality of data nodes comprised by the target index group; selecting M data nodes with the least number of stored documents from the plurality of data nodes comprised by the target index group.

4. The method of claim 1, wherein, The selecting a first data node from the M data nodes according to load information of the M data nodes comprises: selecting a first data node from the M data nodes according to a number of documents corresponding to the M data nodes, network request delay and load information.

5. The method of claim 4, wherein, The selecting a first data node from the M data nodes according to a number of documents corresponding to the M data nodes, network request delay and load information comprises: determining a document value of each data node according to a number of documents stored by the each data node; determining a network delay value of the each data node according to network request delay corresponding to the each data node; determining a load value of the each data node according to load information corresponding to the each data node; determining an accumulated value of the each data node by determining a sum of the document value, the network delay value and the load value of the each data node; selecting a first data node corresponding to a minimum accumulated value from the M data nodes.

6. The method of claim 5, wherein, The determining a network delay value of the each data node according to network request delay corresponding to the each data node comprises: determining a network delay weight corresponding to each data node according to a network request delay corresponding to each data node; determining a network delay value of each data node according to the network delay weight and the network request delay corresponding to each data node.

7. The method of claim 5, wherein, The method further comprises: determining a load weight corresponding to each data node according to load information corresponding to each data node; determining a load value of each data node according to the load weight and the load information corresponding to each data node.

8. The method of claim 1, wherein, The method further comprises: storing the target batch write request to a write cache queue.

9. The method of claim 8, wherein, The method further comprises: in the case of failure of a second data node, obtaining an uncompleted batch write request corresponding to the second data node from the write cache queue, to obtain at least one batch write request; sending the at least one batch write request to the coordination node, the at least one batch write request being used for the coordination node to send a corresponding batch write request to a corresponding client.

10. A distributed search engine system, characterized by The system comprises a client, a master node, a coordination node and a plurality of index groups, each index group comprising a plurality of data nodes. The client is configured to send a target batch write request to the coordination node, the target batch write request comprising a target index and a plurality of document data. The coordination node is configured to send the target batch write request to the master node. The master node is configured to select a target index group from the plurality of index groups according to the target index, select M data nodes from a plurality of data nodes included in the target index group, select one data node from the M data nodes according to load information of the M data nodes, to obtain a first data node, and send node information of the first data node to the coordination node, M being an integer greater than 1. The coordination node is further configured to send the target batch write request to the first data node according to the node information. The first data node is configured to write the plurality of document data to a primary shard.

Images