A method and apparatus for accelerating Elasticsearch search

Abstract

This invention provides a method and apparatus for accelerating Elasticsearch searches. The method includes: S1, adding time metadata to all Elasticsearch index data and saving the time metadata to a Redis database to generate an index list; S2, obtaining the log time range to be searched by the user based on the client request, and determining whether it is a synchronous or asynchronous query; S3, if it is a synchronous query, searching the index list and obtaining a synchronous accurate index list through syntax calculation; S4, if it is an asynchronous query, obtaining an asynchronous accurate index list through combination; S5, returning the query results to the client. This invention improves the efficiency of searching massive logs and reduces the frequency of slow queries and query timeouts by constructing detailed time metadata information.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically to a method and apparatus for accelerating Elasticsearch searches. Background Technology

[0002] Elasticsearch is a distributed, multi-tenant, real-time search and data analytics full-text search engine. By deploying a multi-node Elasticsearch cluster, users can easily and effectively manage petabytes of logs. For massive log volumes, Elasticsearch implements a hot, warm, and cold data storage architecture through index lifecycle management. Under this architecture, we can generally quickly search for recent logs using index aliases. However, as the query time range expands, the number of logs matching the criteria increases rapidly. This is because when Elasticsearch performs distributed retrieval, it queries all indexes under the same alias. If searching still relies on index aliases, slow queries or even timeouts may occur.

[0003] In view of this, the present invention is hereby proposed. Summary of the Invention

[0004] In view of this, the present invention discloses a method and apparatus for accelerating Elasticsearch search, which improves the efficiency of searching massive logs and reduces the frequency of slow queries and query timeouts by constructing detailed time metadata information.

[0005] Specifically, the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention discloses a method for accelerating Elasticsearch searches, comprising the following steps:

[0007] S1. Add time metadata to all index data in Elasticsearch and save the time metadata to a Redis database to generate an index list;

[0008] S2. Based on the client request, obtain the log time range that the user wants to search, and determine whether it is a synchronous or asynchronous query;

[0009] S3. If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list and calculate the synchronous miss_list that does not meet the time range condition. Calculate the synchronous accurate index list through syntax.

[0010] S4. If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively. Add the future index feature_list, and obtain the asynchronous accurate index list by combination.

[0011] S5. Return the query results to the client.

[0012] The above method adds time-based metadata to all Elasticsearch index data, creating more detailed cluster index metadata. When users perform queries, they can retrieve the index list based on the query time range and then use the index list to calculate and obtain the accurate index list, significantly improving the search efficiency of massive logs.

[0013] Furthermore, in step S1, the method for adding the time metadata information includes:

[0014] Create a new periodic task and run it to query and analyze all the index data in the current Elasticsearch cluster; obtain the maximum and minimum values ​​of the date field of each index data through bucket aggregation and add them to the time metadata information.

[0015] Specifically, the Elasticsearch _cat / indices interface is used to obtain a list of all current indexes in the cluster; the _cat / aliases interface is used to obtain the alias and is_write_index information for each index; and the max_bucket and min_bucket aggregations are used to obtain the maximum and minimum values ​​of the date field for each index, max_date and min_date. Note that for indexes where is_write_index is true, max_date does not need to be calculated because the latest logs in that index are still being written continuously.

[0016] Further, in step S1, the saving method includes:

[0017] By utilizing the Redis hash structure, the index alias is set as the Redis key, the actual index name is set as the hash key, and the specific metadata information is set as the hash value.

[0018] Furthermore, in step S3, the method for calculating the syntax includes: using the Elasticsaerch multi-target syntax, subtracting the synchronization miss_list that does not meet the time range condition from the entire index list to obtain the synchronization accurate index list.

[0019] Specifically, the miss_list of indexes that do not meet the time conditions is calculated based on time_range, min_date, and max_date. Using Elasticsaerch's multi-target syntax, the accurate range of the synchronized indexes is the total index list minus the miss_list of indexes that do not meet the conditions (real_list = all_list - miss_list).

[0020] Further, in step S4, the method for combined calculation includes:

[0021] Based on the asynchronous miss_list that does not meet the time range condition: subtract the asynchronous miss_list that does not meet the time range condition from the entire index list to obtain the asynchronous accurate index list 1;

[0022] Alternatively, based on the asynchronous hit_list that meets the time range condition: combine the asynchronous hit_list that meets the time range condition with the future index feature_list to obtain the asynchronous accurate index list 2.

[0023] Furthermore, in step S4, the future index feature_list is a list of new indexes that meet the criteria that are rolled out within the statistical task interval.

[0024] The above-mentioned addition of future index feature_list can estimate future index time metadata, build more complete cluster index metadata information, ensure that no data is missed, and improve data search efficiency.

[0025] Secondly, the present invention discloses an apparatus for accelerating Elasticsearch searches, comprising:

[0026] Time metadata module: Adds time metadata information to all index data, saves the time metadata information to the Redis database, and generates an index list;

[0027] Judgment module: Based on the client request, obtain the time range of the logs that the user wants to search, and determine whether it is a synchronous query or an asynchronous query;

[0028] Synchronous query module: If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list, calculate the synchronous miss_list that does not meet the time range condition, and obtain the synchronous accurate index list through syntax calculation;

[0029] Asynchronous query module: If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively, add the future index feature_list, and obtain the asynchronous accurate index list by combination;

[0030] Result return module: Returns the query results to the client.

[0031] Thirdly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for accelerating Elasticsearch search as described in the first aspect.

[0032] Fourthly, the present invention discloses a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for accelerating Elasticsearch search as described in the first aspect.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] This invention improves the efficiency of searching massive logs and significantly reduces the frequency of slow queries and query timeouts by constructing more detailed index-level metadata. Through the search method of this invention, users can flexibly adjust the extended index time metadata information according to their own business needs, estimate the future index list by calculating the time metadata information, and obtain an accurate index list with a smaller range, thereby improving query efficiency. Attached Figure Description

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0036] Figure 1 This is a schematic diagram of the method flow for accelerating Elasticsearch search provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the method for adding and saving time metadata provided in an embodiment of the present invention;

[0038] Figure 3 A flowchart illustrating the method for obtaining an accurate index list provided in this embodiment of the invention;

[0039] Figure 4 A schematic diagram of a device for accelerating Elasticsearch search provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. However, those skilled in the art will understand that the embodiments described below are some embodiments of the present invention, but not all embodiments, and are only used to illustrate the present invention, and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] To more clearly illustrate the technical solutions in this invention, specific embodiments are described below.

[0043] Example

[0044] Reference Figure 1 As shown, this invention discloses a method for accelerating Elasticsearch searches, comprising the following steps:

[0045] S1. Add time metadata to all index data in Elasticsearch and save the time metadata to a Redis database to generate an index list;

[0046] S2. Based on the client request, obtain the log time range that the user wants to search, and determine whether it is a synchronous or asynchronous query;

[0047] S3. If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list and calculate the synchronous miss_list that does not meet the time range condition. Calculate the synchronous accurate index list through syntax.

[0048] S4. If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively. Add the future index feature_list, and obtain the asynchronous accurate index list by combination.

[0049] S5. Return the query results to the client.

[0050] See Figure 2The method for adding and saving time metadata according to an embodiment of the present invention includes the following steps:

[0051] After the program starts, a periodic task is initiated, with an execution cycle of up to one hour. The task retrieves a list of all current indexes in the Elasticsearch cluster using the `_cat / indices` interface and obtains the alias and `is_write_index` information for each index using the `_cat / aliases` interface. The task aggregates the `date` field values ​​(`max_date` and `min_date`) of each index using `max_bucket` and `min_bucket`, and adds time-based metadata to all index data. Note that indexes with `is_write_index` set to `true` do not need to have their `max_date` calculated, as the latest logs for those indexes are still being continuously written. Finally, a Redis hash structure is used to store the index metadata, where the Redis key is `IndexMeta_index_alias`, the hash key is the actual index name, and the hash value is the specific index metadata.

[0052] See Figure 3 The method for obtaining an accurate index list in this embodiment is shown below, and includes the following steps:

[0053] Receive client requests and obtain the requested date filter condition, date_range; determine whether it is a synchronous or asynchronous query based on the HTTP request URL or HTTP request parameters;

[0054] If it is a synchronous query, the Elasticsearch_msearch interface is used to implement synchronous search. Specifically, the index list stored in Redis can be queried, and the synchronous miss_list that does not meet the time range conditions can be calculated based on time_range, min_date, and max_date. The Elasticsearch multi-target syntax is then used to query and calculate the synchronous accurate index list. The specific calculation method is to subtract the synchronous miss_list that does not meet the time range conditions from the entire index list to obtain the synchronous accurate index list.

[0055] For asynchronous queries, the Elasticsearch_async_search interface is used to implement asynchronous search. Specifically, the index list stored in Redis can be queried, and asynchronous hits that meet the time range conditions and asynchronous misses that do not meet the time range conditions can be calculated based on time_range, min_date, and max_date. Because the asynchronous query interface does not support index list parameters in the _msearch method, and there may be situations where the same alias index contains many actual indexes, or the query URI concatenated from the index list exceeds the default length of the HTTP URI, the asynchronous query search method of this invention needs to count the two types of index lists that are composite and do not meet the time range conditions.

[0056] For asynchronous hit_list that meets the time range conditions, there may be scenarios where new indexes that meet the conditions are rolled out within the interval of the statistical task. Depending on the business scenario, add reasonable future indexes feature_list that meet the conditions and combine them into a new asynchronous accurate index list that meets the conditions, that is, a combination of asynchronous hit_list and future index feature_list.

[0057] During the final query, based on the asynchronous miss_list that does not meet the time range condition: subtract the asynchronous miss_list that does not meet the time range condition from the entire index list to obtain the asynchronous accurate index list 1; or based on the asynchronous hit_list that meets the time range condition: combine the asynchronous hit_list that meets the time range condition with the future index feature_list to obtain the asynchronous accurate index list 2. Finally, the asynchronous accurate index list is selected as asynchronous accurate index list 1, asynchronous accurate index list 2 or the original alias.

[0058] The query results are returned to the client.

[0059] In addition, this invention also provides a device for accelerating Elasticsearch searches, such as... Figure 4 As shown, it specifically includes:

[0060] Time metadata module: Adds time metadata information to all index data, saves the time metadata information to the Redis database, and generates an index list;

[0061] Judgment module: Based on the client request, obtain the time range of the logs that the user wants to search, and determine whether it is a synchronous query or an asynchronous query;

[0062] Synchronous query module: If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list, calculate the synchronous miss_list that does not meet the time range condition, and obtain the synchronous accurate index list through syntax calculation;

[0063] Asynchronous query module: If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively, add the future index feature_list, and obtain the asynchronous accurate index list by combination;

[0064] Result return module: Returns the query results to the client.

[0065] The system mainly consists of the five modules mentioned above. By building this system, the goal of parallel operation can be achieved by mounting the same file system at the same time.

[0066] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of each unit, please refer to the previous method implementation examples, which will not be repeated here.

[0067] Figure 5 This is a schematic diagram of the structure of a computer device disclosed in this invention. (Reference) Figure 5 As shown, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor via a communication bus 403 and is used to store computer instructions executable by the processor 401. The processor 401 is used to read computer instructions from the memory 402 to implement the steps of the method for accelerating Elasticsearch search described in the above embodiments.

[0068] For the above-described apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0069] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accelerating Elasticsearch search, characterized in that, Includes the following steps: S1. Add time metadata to all index data in Elasticsearch and save the time metadata to a Redis database to generate an index list; S2. Based on the client request, obtain the log time range that the user wants to search, and determine whether it is a synchronous or asynchronous query; S3. If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list and calculate the synchronous miss_list that does not meet the time range condition. Calculate the synchronous accurate index list through syntax. S4. If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively. Add the future index feature_list, and obtain the asynchronous accurate index list by combination. S5. Return the query results to the client; In step S3, the method for calculating the syntax includes: using the Elasticsaerch multi-target syntax, subtracting the synchronization miss_list that does not meet the time range condition from the entire index list to obtain the synchronization accurate index list; In step S4, the method for obtaining the asynchronous accurate index list by combination includes: Based on the asynchronous miss_list that does not meet the time range condition: subtract the asynchronous miss_list that does not meet the time range condition from the entire index list to obtain the asynchronous accurate index list 1; Alternatively, based on the asynchronous hit_list that meets the time range condition: combine the asynchronous hit_list that meets the time range condition with the future index feature_list to obtain the asynchronous accurate index list 2; In step S4, the future index feature_list is a list of new indexes that meet the criteria that are rolled out within the interval of the statistical task.

2. The method for accelerating Elasticsearch search according to claim 1, characterized in that, In step S1, the method for adding the time metadata information includes: Create a new periodic task and run it to query and analyze all the index data in the current Elasticsearch cluster; obtain the maximum and minimum values ​​of the date field of each index data through bucket aggregation and add them to the time metadata information.

3. The method for accelerating Elasticsearch search according to claim 1, characterized in that, In step S1, the saving method includes: By utilizing the Redis hash structure, the index alias is set as the Redis key, the actual index name is set as the hash key, and the specific metadata information is set as the hash value.

4. An apparatus for accelerating Elasticsearch searches, using the method as described in any one of claims 1-3, characterized in that, include: Time metadata module: Adds time metadata information to all index data, saves the time metadata information to the Redis database, and generates an index list; Judgment module: Based on the client request, obtain the time range of the logs that the user wants to search, and determine whether it is a synchronous query or an asynchronous query; Synchronous query module: If it is a synchronous query, use the Elasticsearch_msearch interface to search the index list, calculate the synchronous miss_list that does not meet the time range condition, and obtain the synchronous accurate index list through syntax calculation; Asynchronous query module: If it is an asynchronous query, use the Elasticsearch_async_search interface to search the index list, and calculate the asynchronous hit_list that meets the time range condition and the asynchronous miss_list that does not meet the time range condition respectively, add the future index feature_list, and obtain the asynchronous accurate index list by combination; Result return module: Returns the query results to the client.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the steps of the method for accelerating Elasticsearch search as described in any one of claims 1-3.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for accelerating Elasticsearch search as described in any one of claims 1-3.

Images