Method and device for optimizing ES query request link

Abstract

The present invention discloses a method and device for optimizing an ES query request link, which relates to the field of big data technology, and can disperse the pressure of coordination nodes, reduce query links, and improve the efficiency of ES queries. The method includes: regularly requesting the server index list query interface, updating the data storage node addresses corresponding to all index shards in the client cache; matching the coordination node addresses located in the same node area as each data storage node based on the data storage node addresses one by one; initiating an ES query request, querying the data storage node addresses corresponding to the ES query request in the cache, and requesting the server with the matched coordination node addresses as the target address. The device applies the method proposed in the above scheme.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for optimizing an ES query request link. Background Art

[0002] For ES query requests, index data is usually randomly distributed on cluster data nodes according to certain rules. Figure 2 Currently, when an open source native client initiates an ES query request, it first requests the coordination node configured by the client, and then requests the actual data storage node through the coordination node. In other words, most traditional ES query request solutions need to involve cross-node regions, which has the following two problems:

[0003] 1. All query requests are forwarded through several fixed coordination nodes, which causes high pressure on the coordination nodes and fails to achieve data isolation.

[0004] 2. Since the coordination node and the data storage node that requests access are mostly not in the same node area, most of them need to be forwarded through a cross-region coordination node, resulting in a longer link. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for optimizing ES query request links, which can disperse the pressure of coordination nodes, reduce query links, and improve the efficiency of ES queries.

[0006] In order to achieve the above object, a first aspect of the present invention provides an ES query request link optimization method, comprising:

[0007] Regularly request the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache;

[0008] Matching the coordination node address located in the same node area as each data storage node based on the one-to-one correspondence of the data storage node addresses;

[0009] An ES query request is initiated to query the data storage node address corresponding to the ES query request in the cache, and the matched coordination node address is used as the target address to request the server.

[0010] Preferably, before periodically requesting the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache, the method further includes:

[0011] At least one coordination node address is pre-specified by means of fixed configuration in the client, so as to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

[0012] Preferably, the method of initiating an ES query request, querying the data storage node address corresponding to the ES query request in the cache, and using the matched coordination node address as the target address to request the server includes:

[0013] Determine whether the ES query request initiated is a single index request;

[0014] If the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and use the matched coordination node address as the target address to request the server when the query result is searched, otherwise use the coordination node address fixedly configured in the client as the target address to request the server;

[0015] If the judgment result is no, the coordinating node address configured in the client is directly used as the target address to request the server.

[0016] Further, after using the matched coordinating node address as the target address to request the server, the method further includes:

[0017] If the server responds abnormally, the client will reuse the fixed coordinating node address as the target address to request the server.

[0018] Optionally, after the server responds abnormally, the method further includes:

[0019] The client asynchronously requests the server's index list query interface, actively requesting to update the data storage node addresses corresponding to all index shards in the client cache.

[0020] Compared with the prior art, the ES request link optimization method provided by the present invention has the following beneficial effects:

[0021] In the ES request link optimization method provided by the present invention, the data storage node addresses corresponding to all index shards are regularly updated in the client cache, and at the same time, the coordination node addresses that correspond one-to-one to each data storage node address and are located in the same node area are matched. Then, the client is used to initiate an ES query request, and the data storage node address corresponding to the ES query request in the query cache is queried, and the matched coordination node address is used as the target address to request the server again.

[0022] It can be seen that the present invention can preferentially use the coordination node located in the same node area as the target data storage node to forward the access to the server according to the initiated ES query request. Compared with the prior art in which all ES query requests use several fixedly configured coordination nodes to coordinate the access to the server, the scheme of the present invention disperses the distribution of coordination nodes and greatly reduces the forwarding pressure of coordination nodes. In addition, since the coordination node and the target data storage node are located in the same node area, the forwarding times of the coordination node can also be reduced, the request link can be shortened, and data isolation can be achieved better.

[0023] A second aspect of the present invention provides an ES query request link optimization device, which is applied to the ES request link optimization method described in the above technical solution, and the device includes:

[0024] A cache update unit, used to periodically request the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache;

[0025] An association matching unit, configured to match a coordination node address located in the same node area as each data storage node based on the data storage node addresses in a one-to-one correspondence;

[0026] The request query unit is used to initiate an ES query request, query the data storage node address corresponding to the ES query request in the cache, and use the matched coordination node address as the target address to request the server.

[0027] Preferably, it also includes:

[0028] A fixed configuration unit is used to pre-specify at least one coordination node address by means of fixed configuration in the client, and to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

[0029] Preferably, it also includes:

[0030] A judgment unit is used to judge whether the initiated ES query request is a single index request; if the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and when the query result is searched, use the matched coordination node address as the target address to request the server, otherwise use the coordination node address fixedly configured in the client as the target address to request the server; if the judgment result is no, directly use the coordination node address fixedly configured in the client as the target address to request the server.

[0031] Preferably, it also includes:

[0032] The exception handling unit is used to reuse the coordinating node address configured in the client as the target address to request the server if the server responds abnormally.

[0033] Compared with the prior art, the beneficial effects of the ES query request link optimization device provided by the present invention are the same as the beneficial effects of the ES query request link optimization method provided by the above technical solution, which will not be elaborated here.

[0034] A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned ES query request link optimization method are executed.

[0035] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the ES query request link optimization method provided by the above technical solution, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0037] Figure 1 A schematic diagram of a flow chart of an ES query request link optimization method according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of a traditional ES query request link in an embodiment of the present invention;

[0039] Figure 3 Another schematic diagram of a flow chart of an ES query request link optimization method according to an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of the optimized ES query request link in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0042] Embodiment 1

[0043] See also Figure 1 , Figure 3 and Figure 4 This embodiment provides an ES query request link optimization method, including:

[0044] Periodically request the server index list query interface to update the data storage node addresses corresponding to all index shards in the client cache; match the coordination node addresses that are located in the same node area as each data storage node based on the one-to-one correspondence between the data storage node addresses; initiate an ES query request to query the data storage node address corresponding to the ES query request in the cache, and request the server with the matched coordination node address as the target address.

[0045] In the ES request link optimization method provided in this embodiment, the data storage node addresses corresponding to all index shards are regularly updated in the client cache, and the coordination node addresses that correspond one-to-one to each data storage node address and are located in the same node area are matched. Then, the client is used to initiate an ES query request, and the data storage node address corresponding to the ES query request in the query cache is queried, and the matched coordination node address is used as the target address to request the server again.

[0046] It can be seen that this embodiment can preferentially use the coordination node located in the same node area as the target data storage node to forward the access to the server according to the initiated ES query request. Compared with the prior art in which all ES query requests use several fixedly configured coordination nodes to coordinate the access to the server, this embodiment disperses the distribution of the coordination nodes and greatly reduces the forwarding pressure of the coordination nodes. In addition, since the coordination node and the target data storage node are located in the same node area, the number of forwarding times of the coordination node can be reduced, the request link can be shortened, and data isolation can be achieved better.

[0047] In the above embodiment, before periodically requesting the server index list query interface to update the data storage node addresses corresponding to all index fragments in the client cache, the following steps are also included:

[0048] At least one coordination node address is pre-specified by means of fixed configuration in the client, so as to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

[0049] In specific implementation, at least one coordinating node address, that is, the address of CoordinatingNode, is fixedly configured in the client. These fixedly configured coordinating nodes are used when the data storage node address corresponding to the ES query request cannot be queried in the cache, that is, when the coordinating node in the same node area cannot be matched. In order to ensure the normal response of the server, the traditional ES request strategy continues to be used, that is, the fixedly configured coordinating node address is used as the target address to request the server.

[0050] In addition, after the coordination node address is fixedly configured in the client, the client thread starts running, starts a periodic scheduled task to request the server index list query interface, and regularly updates the data storage node addresses corresponding to all index shards in the client cache. The purpose of periodic updating of the client cache is to keep the coordination node address always up to date, avoiding the problem of link extension caused by having to use the traditional solution due to failure to update in time, that is, using a fixed configuration coordination node to access the server.

[0051] In the above embodiment, the method of querying the data storage node address corresponding to the ES query request in the cache and using the matched coordination node address as the target address to request the server includes:

[0052] Determine whether the initiated ES query request is a single index request; if the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and when the query result is searched, use the matched coordination node address as the target address to request the server, otherwise use the coordination node address fixedly configured in the client as the target address to request the server; if the judgment result is no, directly use the coordination node address fixedly configured in the client as the target address to request the server.

[0053] For specific implementation, please refer to Figure 3 In the process of the ES query request initiated by the client, it is necessary to judge whether the current request is a single-index request. If it is a single-index request, the optimized link solution in this embodiment is used to access the server. If it is not a single-index request, the traditional solution is used to access the server. Of course, even if it is a single-index request, there may be a situation where the data storage node address corresponding to the ES query request cannot be found in the client cache. At this time, in order to ensure the normal response of the request, the traditional solution can still be used to access the server. It should be noted that there are various situations that lead to the inability to find the data storage node address corresponding to the ES query request in the client cache. For example, the data storage node address in the client cache is not the latest.

[0054] Generally speaking, a single-index request means that the query results only involve a single data table, and the data in the single data table may be distributed in multiple data storage nodes. A non-single-index request means that the query results involve multiple data tables.

[0055] In the above embodiment, after using the matched coordinating node address as the target address to request the server, the following step is further included:

[0056] If the server responds abnormally, the client will reuse the fixed coordinating node address as the target address to request the server.

[0057] In specific implementation, when the server responds abnormally, the client jumps to the traditional solution process to execute the request and exception handling retry action, that is, using the configured Coordinating Node address as the target address to request the server and wait for the server to respond.

[0058] In the above embodiment, after the server responds abnormally and reuses the fixedly configured coordinating node address in the client as the target address to request the server, the method further includes:

[0059] The client asynchronously requests the server's index list query interface, actively requesting to update the data storage node addresses corresponding to all index shards in the client cache.

[0060] In the specific implementation, the client asynchronously initiates a request to the server's index list query interface to obtain the data storage node addresses corresponding to all index shards, and updates the results to the client's cache.

[0061] In summary, the above embodiments have the following effects:

[0062] 1. It breaks up the request pressure of fixed configuration coordination nodes in traditional solutions and distributes the request pressure to all coordination nodes in the cluster;

[0063] 2. Single-index queries only affect the node where the index is located, reducing the mutual impact of queries between indexes;

[0064] 3. The accessed coordination node and data storage node are in the same node area, which reduces the forwarding of the request link.

[0065] Embodiment 2

[0066] This embodiment provides an ES query request link optimization device, including:

[0067] A cache update unit, used to periodically request the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache;

[0068] An association matching unit, configured to match a coordination node address located in the same node area as each data storage node based on the data storage node addresses in a one-to-one correspondence;

[0069] The request query unit is used to initiate an ES query request, query the data storage node address corresponding to the ES query request in the cache, and use the matched coordination node address as the target address to request the server.

[0070] Preferably, it also includes:

[0071] A fixed configuration unit is used to pre-specify at least one coordination node address by means of fixed configuration in the client, and to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

[0072] Preferably, it also includes:

[0073] A judgment unit is used to judge whether the initiated ES query request is a single index request; if the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and when the query result is searched, use the matched coordination node address as the target address to request the server, otherwise use the coordination node address fixedly configured in the client as the target address to request the server; if the judgment result is no, directly use the coordination node address fixedly configured in the client as the target address to request the server.

[0074] Preferably, it also includes:

[0075] The exception handling unit is used to reuse the coordinating node address configured in the client as the target address to request the server if the server responds abnormally.

[0076] Compared with the prior art, the beneficial effects of the ES query request link optimization device provided in the embodiment of the present invention are the same as the beneficial effects of the ES query request link optimization method provided in the above-mentioned embodiment 1, and are not elaborated here.

[0077] Embodiment 3

[0078] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned ES query request link optimization method are executed.

[0079] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as the beneficial effects of the ES query request link optimization method provided by the above technical solution, and will not be elaborated here.

[0080] A person skilled in the art can understand that all or part of the steps in the above-mentioned inventive method can be implemented by instructing related hardware through a program, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, optical disk, memory card, etc.

[0081] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing an ES query request link, characterized in that: include: Regularly request the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache; Matching a coordination node address located in the same node area as each of the data storage nodes based on the one-to-one correspondence of the data storage node addresses; Initiate an ES query request, and determine whether the initiated ES query request is a single index request, wherein the single index request means that the query result only involves a single data table; If the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and after searching the query result, use the matched coordination node address as the target address to request the server.

2. The method according to claim 1, characterized in that Before periodically requesting the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache, it also includes: At least one coordination node address is pre-specified in a fixed configuration in the client, so as to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

3. The method according to claim 2, characterized in that Initiating an ES query request, and determining whether the initiated ES query request is a single index request further includes: If the judgment result is no, the coordinating node address configured in the client is directly used as the target address to request the server.

4. The method according to any one of claims 1 to 3, characterized in that: After using the matched coordinating node address as the target address to request the server, the method further includes: If the server responds abnormally, the client will reuse the fixed coordinating node address as the target address to request the server.

5. The method according to claim 4, characterized in that When the server responds abnormally, the method further includes: reusing the fixed coordinating node address configured in the client as the target address to request the server; The client asynchronously requests the server's index list query interface, actively requesting to update the data storage node addresses corresponding to all index shards in the client cache.

6. An ES query request link optimization device, characterized in that: include: A cache update unit, used to periodically request the server-side index list query interface to update the data storage node addresses corresponding to all index shards in the client cache; An association matching unit, used for matching the coordination node address located in the same node area as each data storage node based on the data storage node address in a one-to-one correspondence, wherein a single index request is a query result involving only a single data table; A request query unit, used to initiate an ES query request and determine whether the initiated ES query request is a single index request; If the judgment result is yes, continue to query the data storage node address corresponding to the ES query request in the cache, and after searching the query result, use the matched coordination node address as the target address to request the server.

7. The device according to claim 6, characterized in that Also includes: A fixed configuration unit is used to pre-specify at least one coordination node address by means of fixed configuration in the client, and to use the fixedly configured coordination node address as the target address when the data storage node address corresponding to the ES query request cannot be found in the cache.

8. The device according to claim 7, characterized in that Also includes: A judging unit, used to judge whether the ES query request initiated is a single index request; If the judgment result is no, the coordinating node address configured in the client is directly used as the target address to request the server.

9. The device according to claim 8, characterized in that Also includes: The exception handling unit is used to reuse the coordinating node address configured in the client as the target address to request the server if the server responds abnormally.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are executed.

Images