Data Search Method and Device Based on Elasticsearch

Abstract

The present application discloses a data search method and apparatus based on Elasticsearch, which relates to the field of computer technology. The method includes: if the index to be queried is an index in the first index state, search for the first data from the Elasticsearch memory and store the first data in the buffer area; if the index to be queried is an index in the second index state, cache the data stored in the index to be queried into the Elasticsearch memory, and perform a search from the Elasticsearch memory according to the data search request and the index to be queried to obtain the second data, and store the second data in the buffer area; determine the search result of the data search request from the data in the buffer area. Through the present application, the problem in the related art that data cannot be searched out due to excessive memory occupation or even memory exhaustion during data search is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly, to a data search method and apparatus based on Elasticsearch. Background Art

[0002] Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable a large amount of data to have the capabilities of search, analysis, and exploration. Elasticsearch, as a database, can provide functions such as real-time query, search, and aggregation. However, as the amount of data stored in Elasticsearch increases, the amount of disk storage occupied by the data becomes larger and larger, and the memory space required for data retrieval will also become larger and larger. When the data volume exceeds a certain threshold, the large amount of memory space required for ES to retrieve data may cause the retrieval of data to be very slow, and may even exhaust the memory, resulting in the inability to perform normal data indexing and retrieval.

[0003] Regarding the problem in the related art that when performing data search based on Elasticsearch, no effective solution has been proposed yet due to excessive memory occupation or even memory exhaustion, which leads to the inability to search out data. Summary of the Invention

[0004] The main objective of this application is to provide a data search method and apparatus based on Elasticsearch to solve the problem in the related art that when performing data search based on Elasticsearch, no data can be searched out due to excessive memory occupation or even memory exhaustion.

[0005] To achieve the above object, according to one aspect of the present application, a data search method based on Elasticsearch is provided. The method includes: determining a to-be-query index according to a data search request, and determining the index status of the to-be-query index, where the index status is one of the following: a first index status and a second index status, the data stored in the index in the first index status has been cached in the Elasticsearch memory, and the data stored in the index in the second index status has not been cached in the Elasticsearch memory; if the to-be-query index is an index in the first index status, searching for first data from the Elasticsearch memory and storing the first data in a buffer; if the to-be-query index is an index in the second index status, caching the data stored in the to-be-query index in the Elasticsearch memory, and searching from the Elasticsearch memory according to the data search request and the to-be-query index to obtain second data, and storing the second data in the buffer; determining the search result of the data search request from the data in the buffer.

[0006] Further, determining the to-be-query index according to the data search request includes: determining a search time range and attribute information of the to-be-searched data according to the data search request; determining the to-be-query index according to the search time range and the attribute information of the to-be-searched data.

[0007] Further, if the to-be-query index is an index in the first index status, searching for first data from the Elasticsearch memory includes: converting the search time range corresponding to the data search request to obtain a timestamp range; searching for the first data from the Elasticsearch memory according to the timestamp range and the to-be-query index.

[0008] Further, before determining the to-be-query index according to the data search request, the method further includes: creating an index at a preset time period and storing data within a preset time range in the index, where the index is named according to the creation date of the index and the data type stored in the index.

[0009] Further, after storing the data within the preset time range in the index, one of the following methods is used to determine whether to update the index status of the index: setting an occupancy threshold of the Elasticsearch memory and a threshold number of indexes in the first index status, and determining whether to update the index status according to the occupancy threshold and the threshold number; setting a creation time threshold and determining whether to update the index status according to the creation time threshold.

[0010] Further, determining whether to update the index status of the index according to the occupancy threshold and the number of thresholds includes: detecting the occupancy ratio of the Elasticsearch memory and the number of indexes currently in the first index status; if the occupancy ratio is greater than or equal to the occupancy threshold, updating the index status of the first target index to the second index status, and deleting the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold or the number of indexes currently in the first index status is equal to the number of thresholds, where the creation time of the first target index is earlier than that of the remaining indexes.

[0011] Further, determining whether to update the index status of the index according to the creation time threshold includes: counting the creation times of the indexes currently in the first index status; determining whether the difference between the creation time of the indexes currently in the first index status and the current time is greater than or equal to the creation time threshold; if there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, updating the index status of the second target index to the second index status, and deleting the data corresponding to the second target index in the Elasticsearch memory.

[0012] Further, after storing the second data in the buffer area, the method further includes: deleting the data corresponding to the index to be queried in the Elasticsearch memory.

[0013] To achieve the above object, according to another aspect of the present application, a data search device based on Elasticsearch is provided. The device includes: a first determination unit for determining a to-be-query index according to a data search request and determining the index status of the to-be-query index, where the index status is one of the following: a first index status and a second index status, the data stored in the index of the first index status has been cached in the Elasticsearch memory, and the data stored in the index of the second index status has not been cached in the Elasticsearch memory; a first search unit for, if the to-be-query index is an index in the first index status, searching for first data from the Elasticsearch memory and storing the first data in a buffer area; a second search unit for, if the to-be-query index is an index in the second index status, caching the data stored in the to-be-query index in the Elasticsearch memory, and performing a search from the Elasticsearch memory according to the data search request and the to-be-query index to obtain second data and storing the second data in the buffer area; a second determination unit for determining the search result of the data search request from the data in the buffer area.

[0014] Further, the first determination unit includes: a first determination module for determining a search time range and attribute information of data to be searched according to the data search request; a second determination module for determining the to-be-query index according to the search time range and the attribute information of the data to be searched.

[0015] Further, the first search unit includes: a conversion module for converting the search time range corresponding to the data search request to obtain a timestamp range; a search module for searching for the first data from the Elasticsearch memory according to the timestamp range and the to-be-query index.

[0016] Further, the device further includes: a building unit for building an index at a preset time period before determining the to-be-query index according to the data search request and storing data within a preset time range in the index, where the index is named according to the creation date of the index and the data type stored in the index.

[0017] Further, after storing the data within the preset time range into the index, one of the following methods is adopted to determine whether to update the index status of the index: a first setting unit, configured to set an occupancy threshold of the Elasticsearch memory and a threshold number of indexes in a first index status, and determine whether to update the index status of the index according to the occupancy threshold and the threshold number; a second setting unit, configured to set a creation time threshold, and determine whether to update the index status of the index according to the creation time threshold.

[0018] Further, the first setting unit includes: a detection module, configured to detect an occupancy ratio of the Elasticsearch memory and a number of indexes currently in the first index status; a first update module, configured to, if the occupancy ratio is greater than or equal to the occupancy threshold, update the index status of a first target index to a second index status, and delete data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold or the number of indexes currently in the first index status is equal to the threshold number, where a creation time of the first target index is earlier than that of the remaining indexes.

[0019] Further, the second setting unit includes: a statistics module, configured to count creation times of indexes currently in the first index status; a judgment module, configured to judge whether a difference between a creation time of an index currently in the first index status and the current time is greater than or equal to the creation time threshold; a second replacement module, configured to, if there is a second target index whose difference between the creation time and the current time is greater than or equal to the creation time threshold among indexes currently in the first index status, update the index status of the second target index to the second index status, and delete data corresponding to the second target index in the Elasticsearch memory.

[0020] Further, the apparatus further includes: a deletion unit, configured to delete data corresponding to the index to be queried in the Elasticsearch memory after storing the second data in the buffer area.

[0021] To achieve the above object, according to one aspect of the present application, a processor is provided, and the processor is configured to run a program, where the program, when running, executes the Elasticsearch-based data search method described in any one of the above.

[0022] To achieve the above object, according to one aspect of the present application, an electronic device is provided, and the electronic device includes one or more processors and a memory, and the memory is configured to store the Elasticsearch-based data search method described in any one of the above implemented by the one or more processors.

[0023] Through the present application, the following steps are adopted: determining an index to be queried according to a data search request, and determining the index status of the index to be queried, where the index status is one of the following: a first index status and a second index status. The data stored in the index with the first index status has been cached in the Elasticsearch memory, and the data stored in the index with the second index status has not been cached in the Elasticsearch memory; if the index to be queried is an index with the first index status, searching for first data from the Elasticsearch memory and storing the first data in a buffer; if the index to be queried is an index with the second index status, caching the data stored in the index to be queried in the Elasticsearch memory, and performing a search from the Elasticsearch memory according to the data search request and the index to be queried to obtain second data, and storing the second data in the buffer; determining the search result of the data search request from the data in the buffer, which solves the problem in the related art that when performing data search based on Elasticsearch, no data can be searched out due to too much memory occupation or even memory exhaustion. In this solution, the indexes are divided into a first index status and a second index status. For the index with the second index status, it is usually in an unloaded state and does not occupy the Elasticsearch memory space. When retrieving the data stored in the second index status, the data of the index corresponding to the second index status is loaded into the Elasticsearch memory, and the corresponding data is retrieved after loading. After the retrieval, the loading process is removed to release the memory space, avoiding the need to increase the memory space to implement data retrieval, and thus achieving the effect of reducing the cost required for increasing the memory. Description of the Drawings

[0024] The drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0025] Figure 1 is a flowchart of a data search method based on Elasticsearch according to an embodiment of this application;

[0026] Figure 2 is a flowchart of an optional data search method based on Elasticsearch according to an embodiment of this application;

[0027] Figure 3 is a schematic diagram of a data search device based on Elasticsearch according to an embodiment of this application;

[0028] Figure 4It is a schematic diagram of an electronic device provided according to an embodiment of the present application. Detailed implementation manners

[0029] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances for the embodiments of the present application described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is provided between the present system and relevant users or institutions. Before obtaining relevant information, a request for acquisition needs to be sent to the aforementioned users or institutions through the interface, and after receiving the consent information feedback from the aforementioned users or institutions, the relevant information is obtained.

[0033] Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to have the capabilities of search, analysis, and retrieval. Currently, it is widely used in e-commerce product search, App search, enterprise internal information search, IT system search, etc. Most of these products require the storage and query of massive amounts of data, and the data complexity during query is very high, which means that as the amount of data grows, the memory space required for data retrieval will become larger and larger. When storing the same amount of data and providing the same retrieval function, saving and releasing memory becomes particularly important. Therefore, based on the above technical background, a data search method based on Elasticsearch is proposed.

[0034] The present invention will be described below in conjunction with the preferred implementation steps. Figure 1 It is a flowchart of a data search method based on Elasticsearch provided according to an embodiment of the present application, as Figure 1 shown. The method includes the following steps:

[0035] Step S101, determine the index to be queried according to the data search request, and determine the index status of the index to be queried, where the index status is one of the following: the first index status and the second index status. For the first index status, the data stored in the index has been cached in the Elasticsearch memory, and for the second index status, the data stored in the index has not been cached in the Elasticsearch memory.

[0036] Specifically, receive the data search request initiated by the user. The data search request includes relevant information about the data that the user needs to query. For example, the user wants to search for the log data on October 11th. Determine the index to be queried according to the data search request. The index to be queried stores the data searched by the user. After determining the index to be queried according to the data search request, determine the index status of the index to be queried.

[0037] The index status includes the first index status and the second index status. The first index status can also be called the hot index status, and the second index status can also be called the cold index status. For the hot index status, the data stored in the index has been cached in the Elasticsearch memory, and for the cold index status, the data stored in the index has not been cached in the Elasticsearch memory. Generally, the indexes of some data that will not be frequently retrieved can be set to the cold index (freeze) status. When storing data for an index in the cold index status, it will not be loaded into the memory of the Elasticsearch virtual machine, so it will not occupy memory space, thus achieving the purpose of releasing the elasticsearch memory.

[0038] Step S102, if the index to be queried is an index in the first index state, search for the first data from the Elasticsearch memory and store the first data in the buffer;

[0039] Specifically, if the index to be queried is in the hot index state, directly search for the first data from the Elasticsearch memory according to the data search request and the index to be queried, and then store the first data in the buffer.

[0040] Step S103, if the index to be queried is an index in the second index state, cache the data stored in the index to be queried into the Elasticsearch memory, and perform a search from the Elasticsearch memory according to the data search request and the index to be queried to obtain the second data, and store the second data in the buffer;

[0041] Specifically, if the index to be queried is an index in the cold index state, first cache the data stored in the index to be queried into the Elasticsearch memory, and then perform a search from the Elasticsearch memory according to the data search request to obtain the second data, and store the second data in the buffer. After obtaining the second data, delete the data corresponding to the index to be queried in the Elasticsearch memory to release the memory space in a timely manner.

[0042] Step S104, determine the search result of the data search request from the data in the buffer.

[0043] Finally, determine the search result of the data search request through the data in the buffer.

[0044] It should be noted that the index to be queried can be one index or multiple indexes. If the index to be queried is multiple indexes, the above steps are executed for each index to implement data retrieval.

[0045] In summary, in this solution, the indexes are divided into the first index state and the second index state. For the indexes in the second index state, they are usually in the unload state and do not occupy the Elasticsearch memory space. When retrieving the data stored in the second index state, the data corresponding to the index in the second index state is loaded into the Elasticsearch memory, and the corresponding data is retrieved after loading. After the retrieval is completed, the loading process is removed to release the memory space, avoiding the need to increase the memory space to implement data retrieval, and thus achieving the effect of reducing the cost required to increase the memory.

[0046] To improve the accuracy of determining the index to be queried, in the data search method based on Elasticsearch provided in the embodiments of the present application, determining the index to be queried according to a data search request includes: determining a search time range and attribute information of the data to be searched according to the data search request; determining the index to be queried according to the search time range and the attribute information of the data to be searched.

[0047] Specifically, a data search request initiated by a user is received, and the data search request includes relevant information about the data that the user needs to query. The search time range that the user wants to search and the attribute information of the data to be searched are determined according to the data search request. For example, the user wants to search for log data from October 11th to October 17th. Then, the index to be queried is determined according to the search time range and the attribute information of the data to be searched. Since when creating an index, the creation date of the index and the data type stored in the index are used to name the index, the index to be queried can be accurately identified through the search time range and the attribute information of the data to be searched.

[0048] To improve the efficiency and accuracy of data detection, in the data search method based on Elasticsearch provided in the embodiments of the present application, if the index to be queried is an index in the first index state, searching for the first data from the Elasticsearch memory includes: converting the search time range corresponding to the data search request to obtain a timestamp range; searching for the first data from the Elasticsearch memory according to the timestamp range and the index to be queried.

[0049] Specifically, when storing data in an index, the timestamp corresponding to the data is marked. Therefore, the search time range corresponding to the data search request is converted into a timestamp range, and then the first data is searched from the Elasticsearch memory according to the timestamp range and the index to be queried.

[0050] To alleviate the problem of excessive memory occupation in Elasticsearch, in the data search method based on Elasticsearch provided in the embodiments of the present application, after storing the second data in the buffer area, it further includes: deleting the data corresponding to the index to be queried in the Elasticsearch memory.

[0051] Specifically, after obtaining the second data, the data corresponding to the index to be queried in the Elasticsearch memory is deleted to release the memory space in a timely manner.

[0052] In an optional embodiment, when there are multiple indexes to be queried in the cold index state, the data retrieval steps are sequentially executed according to the creation time of the indexes to be queried. First, find the index with the earliest creation time among all the indexes in the cold index state, and load the corresponding data into the Elasticsearch memory (load). Then, query the data to be retrieved in this index in the Elasticsearch memory (read), and temporarily store it in the buffer. After that, unload the data corresponding to this index from the Elasticsearch memory, release the memory space, and continue to query the index in the cold index state with the next earliest creation time until all the indexes in the cold index state have gone through the load->read->unload process, and the search for the index data in the cold index state is completed.

[0053] In the data search method based on Elasticsearch provided in the embodiment of the present application, before determining the indexes to be queried according to the data search request, the method further includes: creating an index at a preset time period, and storing the data within a preset time range into the index, where the index is named according to the creation date of the index and the data type stored in the index.

[0054] Specifically, create an index at a preset time period, and then store the data within a preset time range into the index. In an optional embodiment, the data is stored on a daily basis. At zero o'clock every day, an index is created according to the current system time, and the index name is named after the log type + the current date. For example, if the stored data is syslog and the current date is October 3, 2022, then an index named "syslog-2022-10-03" is created at zero o'clock on October 3, 2022, and all the syslog logs on October 3, 2022 are stored in this index.

[0055] In order to improve the accuracy of updating the index state, in the data search method based on Elasticsearch provided in the embodiment of the present application, after storing the data within a preset time range into the index, one of the following methods is used to determine whether to update the index state: set the occupancy threshold of the Elasticsearch memory and the threshold number of indexes in the first index state, and determine whether to update the index state according to the occupancy threshold and the threshold number; set the creation time threshold, and determine whether to update the index state according to the creation time threshold.

[0056] Determining whether to update the index status based on the occupancy threshold and the number of thresholds includes: detecting the occupancy ratio of the Elasticsearch memory and the number of indexes in the first index status; if the occupancy ratio is greater than or equal to the occupancy threshold, updating the index status of the first target index to the second index status, and deleting the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold or the number of indexes in the first index status is equal to the number of thresholds, where the creation time of the first target index is earlier than that of the remaining indexes.

[0057] Determining whether to update the index status based on the creation time threshold includes: counting the creation times of the indexes in the first index status; judging whether the difference between the creation time of the indexes in the first index status and the current time is greater than or equal to the creation time threshold; if there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, updating the index status of the second target index to the second index status, and deleting the data corresponding to the second target index in the Elasticsearch memory.

[0058] Specifically, set the index cold processing plan according to the actual memory size and usage. It is mainly divided into the following two methods:

[0059] (1) Set the occupancy threshold of the Elasticsearch memory and the number of thresholds of the indexes in the first index status, and determine whether to update the index status according to the occupancy threshold and the number of thresholds;

[0060] (2) Set the creation time threshold and determine whether to update the index status according to the creation time threshold.

[0061] For the first method: Set the occupancy threshold of the Elasticsearch memory and the minimum number of thresholds of the indexes in the hot index status. Monitor the occupancy of the Elasticsearch memory in real time / regularly. When the occupancy of the Elasticsearch memory reaches the set threshold, update the index status of the oldest one or more indexes to the cold index status to release the memory space. If the occupancy of the Elasticsearch memory is still not less than the set occupancy threshold, continue to update the index status until the memory occupancy is less than the set threshold or the number of indexes in the hot index status reaches the set number of thresholds.

[0062] For the second method: set the earliest creation time of the index in the hot index state (that is, the creation time threshold mentioned above), such as setting it to 30 days; regularly (such as every day) check the index status to see whether there are indexes in the hot index state earlier than 30 days before the current time. If so, update the index status of these indexes to the cold index state.

[0063] The above solution saves and releases memory space, supports larger-scale data storage and data retrieval without increasing memory space, and greatly reduces the capital and manpower investment caused by increasing memory space.

[0064] In an optional embodiment, the following may be used: Figure 2 The flowchart shown implements data search based on Elasticsearch.

[0065] (1) Find all indexes based on the queried time range and index name (creation time).

[0066] (2) Query the index status of all indexes. If the index is in a hot index state (open state), directly query the corresponding data in the index from Elasticsearch and return it, and temporarily store the queried data in the cache area.

[0067] (3) If the index status is cold (frozen), perform the following operations:

[0068] a) Find the index with the earliest creation time among all frozen indexes and load its corresponding data into memory (load);

[0069] b) Query the data to be retrieved in this index (read) and temporarily store it in the cache;

[0070] c) Unload the data corresponding to the index from the memory to release the memory space;

[0071] d) Remove this index from the indexes to be queried, and continue to query the next frozen state index with the earliest creation time;

[0072] e) Until all frozen indexes have gone through the load->read->unload process, the frozen status index data query is completed;

[0073] (4) Summarize the data queried from all open and frozen state indexes and return it to the querying user;

[0074] (5) This query ends.

[0075] The data search method based on Elasticsearch provided by the embodiments of the present application determines the index to be queried according to a data search request, and determines the index status of the index to be queried, where the index status is one of the following: the first index status and the second index status. The data stored in the index with the first index status has been cached in the Elasticsearch memory, and the data stored in the index with the second index status has not been cached in the Elasticsearch memory. If the index to be queried is an index with the first index status, the first data is searched from the Elasticsearch memory and stored in the buffer area. If the index to be queried is an index with the second index status, the data stored in the index to be queried is cached in the Elasticsearch memory, and based on the data search request and the index to be queried, a search is performed from the Elasticsearch memory to obtain the second data, and the second data is stored in the buffer area. The search result of the data search request is determined from the data in the buffer area, which solves the problem in the related art that when performing data search based on Elasticsearch, data cannot be searched out due to too much memory occupation or even memory exhaustion. In this solution, the index is divided into the first index status and the second index status. For the index with the second index status, it is usually in the unload state and does not occupy the Elasticsearch memory space. When retrieving the data stored in the second index status, the data of the index corresponding to the second index status is loaded into the Elasticsearch memory. After loading, the corresponding data is retrieved, and after the retrieval, the loading process is removed to release the memory space, avoiding realizing data retrieval by increasing the memory space, and thus achieving the effect of reducing the cost required for increasing the memory.

[0076] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0077] The embodiments of the present application also provide a data search device based on Elasticsearch. It should be noted that the data search device based on Elasticsearch in the embodiments of the present application can be used to execute the data search method based on Elasticsearch provided by the embodiments of the present application. The following introduces the data search device based on Elasticsearch provided by the embodiments of the present application.

[0078] Figure 3 is a schematic diagram of the data search device based on Elasticsearch according to the embodiments of the present application. As Figure 3As shown in the figure, the device includes: a first determination unit 301, a first search unit 302, a second search unit 303, and a second determination unit 304.

[0079] The first determination unit 301 is configured to determine an index to be queried according to a data search request, and determine the index status of the index to be queried, where the index status is one of the following: a first index status and a second index status. For the first index status, the data stored in the index has been cached in the Elasticsearch memory. For the second index status, the data stored in the index has not been cached in the Elasticsearch memory.

[0080] The first search unit 302 is configured to, if the index to be queried is an index with the first index status, search for first data from the Elasticsearch memory and store the first data in the buffer.

[0081] The second search unit 303 is configured to, if the index to be queried is an index with the second index status, cache the data stored in the index to be queried in the Elasticsearch memory, and perform a search from the Elasticsearch memory according to the data search request and the index to be queried to obtain second data, and store the second data in the buffer.

[0082] The second determination unit 304 is configured to determine the search result of the data search request from the data in the buffer.

[0083] Specifically, a data search request initiated by a user is received. The data search request includes relevant information about the data that the user needs to query. For example, the user wants to search for log data on October 11th. The index to be queried is determined according to the data search request, and the data searched by the user is stored in the index to be queried. After determining the index to be queried according to the data search request, the index status of the index to be queried is determined.

[0084] The index status includes a first index status and a second index status. The first index status can also be called the hot index status, and the second index status can also be called the cold index status. For the hot index status, the data stored in the index has been cached in the Elasticsearch memory. For the cold index status, the data stored in the index has not been cached in the Elasticsearch memory. Generally, the indexes of some data that will not be frequently retrieved can be set to the cold index (freeze) status. When storing data for an index in the cold index status, it will not be loaded into the memory of the Elasticsearch virtual machine, so it will not occupy memory space, thereby achieving the purpose of releasing the elasticsearch memory.

[0085] If the index to be queried is in the hot index state, directly search for the first data from the Elasticsearch memory according to the data search request and the index to be queried, and then store the first data in the buffer.

[0086] If the index to be queried is an index in the cold index state, first cache the data stored in the index to be queried into the Elasticsearch memory, then search from the Elasticsearch memory according to the data search request to obtain the second data, and store the second data in the buffer. After obtaining the second data, delete the data corresponding to the index to be queried in the Elasticsearch memory to release the memory space in time. Finally, determine the search result of the data search request through the data in the buffer.

[0087] It should be noted that the index to be queried can be a single index or multiple indexes. If the index to be queried is multiple indexes, the above steps are executed for each index to achieve data retrieval.

[0088] In summary, in this solution, the indexes are divided into the first index state and the second index state. For the indexes in the second index state, they are usually in the unload state and do not occupy the Elasticsearch memory space. When retrieving the data stored in the second index state, the data of the index corresponding to the second index state is loaded into the Elasticsearch memory. After loading, the corresponding data is retrieved, and after the retrieval, the loading process is removed to release the memory space, avoiding the need to increase the memory space to achieve data retrieval, thereby achieving the effect of reducing the cost required to increase the memory.

[0089] The data search device based on Elasticsearch provided by the embodiment of the present application, through the first determination unit 301, determines the index to be queried according to the data search request, and determines the index status of the index to be queried, where the index status is one of the following: the first index status and the second index status. The data stored in the index in the first index status has been cached in the Elasticsearch memory, and the data stored in the index in the second index status has not been cached in the Elasticsearch memory; if the index to be queried is the index in the first index status, the first search unit 302 searches for the first data from the Elasticsearch memory and stores the first data in the buffer area; if the index to be queried is the index in the second index status, the second search unit 303 caches the data stored in the index to be queried in the Elasticsearch memory, and searches from the Elasticsearch memory according to the data search request and the index to be queried to obtain the second data, and stores the second data in the buffer area; the second determination unit 304 determines the search result of the data search request from the data in the buffer area, solving the problem in the related art that when performing data search based on Elasticsearch, data cannot be searched out due to too much memory occupation or even memory exhaustion. In this solution, the index is divided into the first index status and the second index status. For the index in the second index status, it is usually in the unload state and does not occupy the Elasticsearch memory space. When retrieving the data stored in the second index status, the data of the index corresponding to the second index status is loaded into the Elasticsearch memory, and the corresponding data is retrieved after loading. After the retrieval is completed, the loading process is removed to release the memory space, avoiding realizing data retrieval by increasing the memory space, and thus achieving the effect of reducing the cost required for increasing the memory.

[0090] Optionally, in the data search device based on Elasticsearch provided by the embodiment of the present application, the first determination unit 301 includes: a first determination module, configured to determine the search time range and the attribute information of the data to be searched according to the data search request; a second determination module, configured to determine the index to be queried according to the search time range and the attribute information of the data to be searched.

[0091] Specifically, a data search request initiated by a user is received, and the data search request includes information related to the data that the user needs to query. Determine the search time range that the user wants to search and the attribute information of the data to be searched according to the data search request. For example, the user wants to search for log data from October 11th to October 17th. Then, determine the index to be queried according to the search time range and the attribute information of the data to be searched. Since when creating an index, the index is named according to the creation date of the index and the data type stored in the index, the index to be queried can be accurately identified through the search time range and the attribute information of the data to be searched.

[0092] Optionally, in the data search device based on Elasticsearch provided in the embodiment of the present application, the first search unit 302 includes: a conversion module, configured to convert the search time range corresponding to the data search request to obtain a timestamp range; a search module, configured to search for the first data from the Elasticsearch memory according to the timestamp range and the index to be queried.

[0093] Specifically, when storing data in an index, the corresponding timestamp of the data will be marked. Therefore, convert the search time range corresponding to the data search request to a timestamp range, and then search for the first data from the Elasticsearch memory according to the timestamp range and the index to be queried.

[0094] Optionally, in the data search device based on Elasticsearch provided in the embodiment of the present application, the device further includes: a creation unit, configured to create an index at a preset time period before determining the index to be queried according to the data search request, and store the data within the preset time range into the index, where the index is named according to the creation date of the index and the data type stored in the index.

[0095] Specifically, create an index at a preset time period, and then store the data within the preset time range into the index. In an optional embodiment, the data is stored in units of days. An index is created at zero o'clock every day according to the current system time, and the index name is named after the log type + the current date. For example, if the stored data is syslog and the current date is October 3, 2022, then an index named "syslog-2022-10-03" is created at zero o'clock on October 3, 2022, and all syslog logs on October 3, 2022 are stored in this index.

[0096] Optionally, in the data search device based on Elasticsearch provided in the embodiments of the present application, after storing the data within a preset time range into the index, one of the following methods is adopted to determine whether to update the index status of the index: The first setting unit is used to set the occupancy threshold of the Elasticsearch memory and the threshold number of indexes in the first index status, and determine whether to update the index status of the index according to the occupancy threshold and the threshold number; The second setting unit is used to set the creation time threshold and determine whether to update the index status of the index according to the creation time threshold.

[0097] Optionally, in the data search device based on Elasticsearch provided in the embodiments of the present application, the first setting unit includes: a detection module, configured to detect the occupancy ratio of the Elasticsearch memory and the number of indexes currently in the first index status; a first update module, configured to, if the occupancy ratio is greater than or equal to the occupancy threshold, update the index status of the first target index to the second index status, and delete the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold or the number of indexes currently in the first index status is equal to the threshold number, where the creation time of the first target index is earlier than that of the remaining indexes.

[0098] Optionally, in the data search device based on Elasticsearch provided in the embodiments of the present application, the second setting unit includes: a statistics module, configured to count the creation times of the indexes currently in the first index status; a judgment module, configured to judge whether the difference between the creation time of the index currently in the first index status and the current time is greater than or equal to the creation time threshold; a second replacement module, configured to, if there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, update the index status of the second target index to the second index status, and delete the data corresponding to the second target index in the Elasticsearch memory.

[0099] Specifically, an index cold processing scheme is set according to the actual memory size and usage. It is mainly divided into the following two methods:

[0100] (1) Set the occupancy threshold of the Elasticsearch memory and the threshold number of indexes in the first index status, and determine whether to update the index status of the index according to the occupancy threshold and the threshold number;

[0101] (2) Set the creation time threshold and determine whether to update the index status of the index according to the creation time threshold.

[0102] For the first method: Set the memory occupancy threshold of Elasticsearch and the minimum number of indexes in the hot index state. Monitor the memory occupancy of Elasticsearch in real time / regularly. When the memory occupancy of Elasticsearch reaches the set threshold, update the index status of one or more indexes with the oldest index creation time to the cold index state to release memory space. If the memory occupancy of Elasticsearch is still not less than the set occupancy threshold, continue to update the index status until the memory occupancy is less than the set threshold or the number of indexes in the hot index state reaches the set threshold number.

[0103] For the second method: Set the earliest creation time of the indexes in the hot index state (i.e., the above-mentioned creation time threshold), such as setting it to 30 days; perform index status checks regularly (such as every day) to check whether there are indexes in the hot index state that are earlier than 30 days before the current time. If there are, then update the index status of these indexes to the cold index state.

[0104] Save and release memory space through the above solution content. Without increasing the memory space, it supports larger-scale data storage and data retrieval, greatly reducing the capital and manpower investment brought by increasing the memory space.

[0105] Optionally, in the data search device based on Elasticsearch provided in the embodiment of the present application, the device further includes: a deletion unit, configured to delete the data corresponding to the index to be queried in the Elasticsearch memory after storing the second data in the buffer area.

[0106] Specifically, after obtaining the second data, delete the data corresponding to the index to be queried in the Elasticsearch memory in time to release memory space.

[0107] In an optional embodiment, when there are multiple indexes to be queried in the cold index state, perform data retrieval steps in sequence according to the creation time of the indexes to be queried. First, find the index with the earliest creation time among all indexes in the cold index state, and load its corresponding data into the Elasticsearch memory (load), then query the data to be retrieved in this index in the Elasticsearch memory (read), and temporarily store it in the buffer area. Then unload the data corresponding to this index from the Elasticsearch memory to release memory space, and continue to query the index in the cold index state with the next earliest creation time until all indexes in the cold index state have gone through the load->read->unload process and the data search of the indexes in the cold index state is completed.

[0108] In an optional embodiment, the following may be used: Figure 2 The flowchart shown implements data search based on Elasticsearch.

[0109] (1) Find all indexes based on the queried time range and index name (creation time).

[0110] (2) Query the index status of all indexes. If the index is in a hot index state (open state), directly query the corresponding data in the index from Elasticsearch and return it, and temporarily store the queried data in the cache area.

[0111] (3) If the index status is cold (frozen), perform the following operations:

[0112] a) Find the index with the earliest creation time among all frozen indexes and load its corresponding data into memory (load);

[0113] b) Query the data to be retrieved in this index (read) and temporarily store it in the cache;

[0114] c) Unload the data corresponding to the index from the memory to release the memory space;

[0115] d) Remove this index from the indexes to be queried, and continue to query the next frozen state index with the earliest creation time;

[0116] e) Until all frozen indexes have gone through the load->read->unload process, the frozen status index data query is completed;

[0117] (4) Summarize the data queried from all open and frozen state indexes and return it to the querying user;

[0118] (5) This query ends.

[0119] The data search device based on Elasticsearch includes a processor and a memory. The above-mentioned first determination unit 301, first search unit 302, second search unit 303 and second determination unit 304 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0120] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and data search based on Elasticsearch can be achieved by adjusting kernel parameters.

[0121] The memory may include non - permanent memory in a computer - readable medium, in the form of random access memory (RAM) and / or non - volatile memory such as read - only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0122] An embodiment of the present invention provides a computer - readable storage medium, on which a program is stored, and when the program is executed by a processor, a data search method based on Elasticsearch is implemented.

[0123] An embodiment of the present invention provides a processor for running a program, wherein when the program runs, a data search method based on Elasticsearch is executed.

[0124] As Figure 4 shown, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, the following steps are implemented: determining a query index to be queried according to a data search request, and determining the index status of the query index, where the index status is one of the following: a first index status and a second index status. For the index with the first index status, the data stored in the index has been cached in the Elasticsearch memory; for the index with the second index status, the data stored in the index has not been cached in the Elasticsearch memory. If the query index is an index with the first index status, searching for first data from the Elasticsearch memory and storing the first data in a buffer area; if the query index is an index with the second index status, caching the data stored in the query index in the Elasticsearch memory, and searching from the Elasticsearch memory according to the data search request and the query index to obtain second data, and storing the second data in the buffer area; determining the search result of the data search request from the data in the buffer area.

[0125] Optionally, determining the query index to be queried according to a data search request includes: determining a search time range and attribute information of the data to be searched according to the data search request; determining the query index according to the search time range and the attribute information of the data to be searched.

[0126] Optionally, if the query index is an index with the first index status, searching for first data from the Elasticsearch memory includes: converting the search time range corresponding to the data search request to obtain a timestamp range; searching for first data from the Elasticsearch memory according to the timestamp range and the query index.

[0127] Optionally, after storing the second data in the buffer area, the method further includes: deleting the data corresponding to the index to be queried in the Elasticsearch memory.

[0128] Optionally, before determining the index to be queried according to the data search request, the method further includes: creating an index at a preset time period, and storing the data within a preset time range into the index, wherein the index is named according to the creation date of the index and the data type stored in the index.

[0129] Optionally, after storing the data within the preset time range into the index, one of the following methods is adopted to determine whether to update the index status of the index: setting the occupancy threshold of the Elasticsearch memory and the threshold number of indexes in the first index status, and determining whether to update the index status of the index according to the occupancy threshold and the threshold number; setting the creation time threshold, and determining whether to update the index status of the index according to the creation time threshold.

[0130] Optionally, determining whether to update the index status of the index according to the occupancy threshold and the threshold number includes: detecting the occupancy ratio of the Elasticsearch memory and the number of indexes currently in the first index status; if the occupancy ratio is greater than or equal to the occupancy threshold, updating the index status of the first target index to the second index status, and deleting the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold, or the number of indexes currently in the first index status is equal to the threshold number, wherein the creation time of the first target index is earlier than that of the remaining indexes.

[0131] Optionally, determining whether to update the index status of the index according to the creation time threshold includes: counting the creation time of the indexes currently in the first index status; judging whether the difference between the creation time of the indexes currently in the first index status and the current time is greater than or equal to the creation time threshold; if there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, updating the index status of the second target index to the second index status, and deleting the data corresponding to the second target index in the Elasticsearch memory.

[0132] The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0133] The present application also provides a computer program product which, when executed on a data processing device, is adapted to execute a program initialized with the following method steps: determining a query index to be queried according to a data search request, and determining the index status of the query index to be queried, where the index status is one of the following: a first index status and a second index status, the data stored in the index of the first index status has been cached in the Elasticsearch memory, and the data stored in the index of the second index status has not been cached in the Elasticsearch memory; if the query index to be queried is an index with the first index status, searching for first data from the Elasticsearch memory and storing the first data in a buffer; if the query index to be queried is an index with the second index status, caching the data stored in the query index in the Elasticsearch memory, and searching from the Elasticsearch memory according to the data search request and the query index to obtain second data, and storing the second data in the buffer; determining the search result of the data search request from the data in the buffer.

[0134] Optionally, determining a query index to be queried according to a data search request includes: determining a search time range and attribute information of the data to be searched according to the data search request; determining the query index to be queried according to the search time range and the attribute information of the data to be searched.

[0135] Optionally, if the query index to be queried is an index with the first index status, searching for first data from the Elasticsearch memory includes: converting the search time range corresponding to the data search request to obtain a time stamp range; searching for first data from the Elasticsearch memory according to the time stamp range and the query index to be queried.

[0136] Optionally, after storing the second data in the buffer, the method further includes: deleting the data corresponding to the query index in the Elasticsearch memory.

[0137] Optionally, before determining a query index to be queried according to a data search request, the method further includes: creating an index at a preset time period and storing the data within a preset time range into the index, where the index is named according to the creation date of the index and the data type stored in the index.

[0138] Optionally, after storing the data within a preset time range into the index, one of the following methods is adopted to determine whether to update the index status of the index: setting an occupancy threshold of the Elasticsearch memory and a threshold number of indexes in the first index status, and determining whether to update the index status of the index according to the occupancy threshold and the threshold number; setting a creation time threshold and determining whether to update the index status of the index according to the creation time threshold.

[0139] Optionally, determining whether to update the index status of an index based on an occupancy threshold and the number of thresholds includes: detecting the occupancy ratio of the Elasticsearch memory and the number of indexes currently in the first index status; if the occupancy ratio is greater than or equal to the occupancy threshold, updating the index status of the first target index to the second index status, and deleting the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold, or the number of indexes currently in the first index status is equal to the number of thresholds, where the creation time of the first target index is earlier than that of the remaining indexes.

[0140] Optionally, determining whether to update the index status of an index based on a creation time threshold includes: counting the creation times of the indexes currently in the first index status; determining whether the difference between the creation time of the indexes currently in the first index status and the current time is greater than or equal to the creation time threshold; if there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, updating the index status of the second target index to the second index status, and deleting the data corresponding to the second target index in the Elasticsearch memory.

[0141] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0143] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks or blocks.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks or blocks.

[0145] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0146] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0147] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0148] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the element.

[0149] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The above are only the embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A data search method based on Elasticsearch, characterized in that, Including: Determine the index to be queried according to the data search request, and determine the index status of the index to be queried, where the index status is one of the following: the first index status and the second index status. The data stored in the index with the first index status has been cached in the Elasticsearch memory, and the data stored in the index with the second index status has not been cached in the Elasticsearch memory; If the index to be queried is an index with the first index status, search for the first data from the Elasticsearch memory and store the first data in the buffer; If the index to be queried is an index with the second index status, cache the data stored in the index to be queried in the Elasticsearch memory, and search from the Elasticsearch memory according to the data search request and the index to be queried to obtain the second data, and store the second data in the buffer; Determine the search result of the data search request from the data in the buffer; Wherein, after storing the second data in the buffer, the method further includes: Delete the data corresponding to the index to be queried in the Elasticsearch memory.

2. The method according to claim 1, wherein Determining the index to be queried according to the data search request includes: Determine the search time range and the attribute information of the data to be searched according to the data search request; Determine the index to be queried according to the search time range and the attribute information of the data to be searched.

3. The method according to claim 2, wherein If the index to be queried is an index with the first index status, searching for the first data from the Elasticsearch memory includes: Convert the search time range corresponding to the data search request to obtain a timestamp range; Search for the first data from the Elasticsearch memory according to the timestamp range and the index to be queried.

4. The method according to claim 1, characterized in that, Before determining the index to be queried according to the data search request, the method further includes: Create an index at a preset time period and store the data within the preset time range in the index, where the index is named according to the creation date of the index and the data type stored in the index.

5. The method according to claim 4, wherein After storing the data within the preset time range in the index, determine whether to update the index status of the index in one of the following ways: Set the occupancy threshold of the Elasticsearch memory and the threshold number of indexes in the first index status, and determine whether to update the index status of the index according to the occupancy threshold and the threshold number; Set the creation time threshold and determine whether to update the index status of the index according to the creation time threshold.

6. The method according to claim 5, wherein Determining whether to update the index status according to the occupancy threshold and the threshold number includes: Detect the occupancy ratio of the Elasticsearch memory and the number of indexes currently in the first index status; If the occupancy ratio is greater than or equal to the occupancy threshold, update the index status of the first target index to the second index status, and delete the data corresponding to the first target index in the Elasticsearch memory until the occupancy ratio is less than the occupancy threshold or the number of indexes in the first index status is equal to the threshold number, where the creation time of the first target index is earlier than that of the remaining indexes.

7. The method according to claim 5, characterized in that, Determining whether to update the index status of an index according to the creation time threshold includes: Counting the creation times of the indexes in the first index status currently; Judging whether the difference between the creation time of the indexes in the first index status currently and the current time is greater than or equal to the creation time threshold; If there is a second target index in the first index status whose difference between the creation time and the current time is greater than or equal to the creation time threshold, update the index status of the second target index to the second index status, and delete the data corresponding to the second target index in the Elasticsearch memory.

8. A data search device based on Elasticsearch, characterized in that, including: A first determination unit, configured to determine a to-be-query index according to a data search request, and determine the index status of the to-be-query index, where the index status is one of the following: a first index status and a second index status, the data stored in the index in the first index status has been cached in the Elasticsearch memory, and the data stored in the index in the second index status has not been cached in the Elasticsearch memory; A first search unit, configured to, if the to-be-query index is an index in the first index status, search for first data from the Elasticsearch memory and store the first data in a buffer; A second search unit, configured to, if the to-be-query index is an index in the second index status, cache the data stored in the to-be-query index in the Elasticsearch memory, and perform a search from the Elasticsearch memory according to the data search request and the to-be-query index to obtain second data, and store the second data in the buffer; A second determination unit, configured to determine a search result of the data search request from the data in the buffer; Wherein, the device further includes: a deletion unit, configured to delete the data corresponding to the to-be-query index in the Elasticsearch memory after storing the second data in the buffer.

9. A processor, characterized in that, The processor is used to run a program, where the program, when running, executes the data search method based on Elasticsearch according to any one of claims 1 to 7.

Images