Data retrieval methods, systems, devices, electronic equipment and storage media

By building candidate indexes locally and matching them with target index values, the target data can be retrieved locally, solving the problem of low retrieval efficiency caused by scattered storage of log data and achieving efficient and accurate data retrieval.

CN114398520BActive Publication Date: 2025-10-31BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202210053020.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-18
Publication Date
2025-10-31
Estimated Expiration
2042-01-18

AI Technical Summary

Technical Problem

In existing technologies, log data is stored across multiple servers, resulting in low data retrieval efficiency, high network overhead, and difficulty in achieving fast and efficient data retrieval.

Method used

Candidate indexes are built locally, and the target index is matched with the candidate index values ​​to retrieve the target data from the local machine, avoiding network connections to read the target data and completing the retrieval and reading directly locally.

Benefits of technology

It reduces network overhead, improves data retrieval efficiency and accuracy, and reduces latency and resource consumption caused by network connections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114398520B_ABST
    Figure CN114398520B_ABST
Patent Text Reader

Abstract

This disclosure provides data retrieval methods, systems, devices, electronic devices, and storage media, relating to fields such as big data processing. A specific implementation scheme includes: receiving a retrieval request from a first device; determining a target index matching the retrieval request based on candidate index values ​​contained in a candidate index; wherein the candidate index is constructed based on locally stored candidate data; obtaining target data from the locally stored candidate data based on the target file offset contained in the target index; and feeding the target data back to the first device as a retrieval result. Embodiments of this disclosure can reduce network overhead and improve retrieval efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of big data processing technology. Background Technology

[0002] With the development of computer technology, the amount of data is increasing rapidly, and users have higher and higher requirements for log retrieval. However, the logs are large in quantity and are stored in multiple servers in a relatively scattered manner. Therefore, how to provide a faster and more efficient data retrieval method has become a problem that needs to be solved. Summary of the Invention

[0003] This disclosure provides a data retrieval method, system, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, a data retrieval method is provided, comprising:

[0005] Receive the search request sent by the first device;

[0006] Based on the candidate index values ​​contained in the candidate index, a target index matching the retrieval request is determined; wherein, the candidate index is constructed based on locally stored candidate data;

[0007] Based on the target file offset contained in the target index, the target data is obtained from the locally stored candidate data;

[0008] The target data is fed back to the first device as a search result.

[0009] According to a second aspect of this disclosure, a data retrieval system is provided, comprising: a data retrieval device and a first apparatus; wherein...

[0010] The data retrieval device is configured to receive a retrieval request from a first device; determine a target index matching the retrieval request based on candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data; obtain target data from the locally stored candidate data based on the target file offset contained in the target index; and feed back the target data as a retrieval result to the first device.

[0011] The first device is used to send a search request to the data retrieval device and receive the search results fed back by the data retrieval device.

[0012] According to a third aspect of this disclosure, a data retrieval apparatus is provided, comprising:

[0013] The receiving module is used to receive retrieval requests sent by the first device;

[0014] An index matching module is used to determine a target index that matches the retrieval request based on the candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data;

[0015] The retrieval module is used to retrieve target data from the locally stored candidate data based on the target file offset contained in the target index;

[0016] The sending module is used to send the target data back to the first device as a retrieval result.

[0017] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0018] At least one processor; and

[0019] The memory is communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the aforementioned retrieval method of the first aspect.

[0021] According to a fifth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the aforementioned method.

[0022] According to a sixth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned method.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0024] The solution provided in this embodiment, upon receiving a search request, determines the target file offset based on the target index value matched by the search request, thereby obtaining the target data, and feeding this target data back to the first device as a search result. In this way, the retrieval and reading of target data can be completed directly locally, avoiding the need to connect to a server storing candidate data via the network for target data reading, reducing network overhead and improving search efficiency. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a schematic flowchart of a data retrieval method according to an embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram showing the location where target data is stored in a target file according to an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of the composition structure of a data retrieval system according to an embodiment of the present disclosure;

[0029] Figure 4 It is based on the ELK cluster architecture diagram;

[0030] Figure 5 This is a schematic diagram of a processing scenario of a data retrieval device in a data retrieval system according to an embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram of a processing scenario of a data retrieval system according to another embodiment of the present disclosure;

[0032] Figure 7 This is a schematic diagram of the composition structure of a data retrieval device according to another embodiment of the present disclosure;

[0033] Figure 8 This is a schematic diagram of another component structure of a data retrieval device according to another embodiment of the present disclosure;

[0034] Figure 9 This is a block diagram of an electronic device used to implement the retrieval method of the embodiments of this disclosure. Detailed Implementation

[0035] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] The first aspect of this disclosure provides a data retrieval method, such as... Figure 1 As shown, it includes:

[0037] S101: Receive a retrieval request from the first device;

[0038] S102: Based on the candidate index values ​​contained in the candidate index, determine the target index that matches the retrieval request; wherein, the candidate index is constructed based on locally stored candidate data;

[0039] S103: Based on the target file offset contained in the target index, obtain the target data from the locally stored candidate data;

[0040] S104: The target data is fed back to the first device as a search result.

[0041] The data retrieval method provided in this embodiment can be applied to a server, and more specifically, to a service agent within the server, such as a log agent. The server can be a server used to store logs. It should be understood that the aforementioned server can be any one of multiple servers in the entire system; that is, the service agent within any of the multiple servers can execute the data retrieval method provided in this embodiment, but this will not be elaborated upon individually.

[0042] The first device can be a client device; the first device can concurrently send the retrieval request to service proxies of multiple servers, and the service proxies of the servers provided in this embodiment can be any one of the service proxies of the multiple servers.

[0043] The candidate index is constructed based on locally stored candidate data.

[0044] The number of candidate data can be one or more. More specifically, any one of the candidate data can be a historical log; the historical log can be obtained based on the user's historical query operations. For example, if user A's historical query operation is: querying weather information at time B; then a historical log can be generated based on this historical query operation.

[0045] The candidate index includes the candidate index value and the candidate file offset. The candidate index value may contain key information in multiple dimensions. The candidate index is in Key-Value (KV) format.

[0046] The number of candidate indexes can be one or more. Each candidate index can contain one or more candidate file offsets. Each of the one or more candidate file offsets indicates the location where candidate data is stored locally.

[0047] The number of target data can be one or more, and this embodiment does not limit it.

[0048] As can be seen, by adopting the above scheme, upon receiving a retrieval request, the target file offset can be determined based on the target index value matched by the retrieval request, thereby obtaining the target data and feeding it back to the first device as a retrieval result. In this way, the retrieval and reading of target data can be completed directly locally, avoiding the need to connect to a server storing candidate data via the network for target data reading, reducing network overhead and improving retrieval efficiency.

[0049] In one implementation, the candidate index value in the candidate index may contain key information in N dimensions, as well as one or more candidate file offsets; N is an integer greater than or equal to 1.

[0050] The key information in the N dimensions may include at least one of the following: candidate window termination timestamp, candidate data related identifier, object related identifier, and candidate data content keywords.

[0051] The candidate window termination timestamp is used to characterize the termination timestamp of the preset time window corresponding to the candidate index. When the candidate index includes the candidate window termination timestamp, the candidate index is constructed based on one or more candidate data with the same keyword within its corresponding preset time window.

[0052] The candidate data-related identifier can specifically be the ID (identity) of the candidate data. More specifically, it can refer to the ID of a particular candidate log, which can be represented as log-ID (i.e., the ID of the log).

[0053] The object-related identifier can specifically be the object's ID; the object can be the querying user, for example, the user's ID.

[0054] The candidate data content keywords can specifically be the content keywords contained in the candidate logs. For example, if user A's historical query operation is: querying weather information at time B; then a historical log can be generated based on this historical query operation; this historical log is the aforementioned candidate data; correspondingly, the candidate data content keyword can be "weather".

[0055] It should be noted that if the candidate index is constructed in different ways, the candidate index value and the candidate file offset contained in the candidate index will also be different. For example, the candidate index can be constructed based on a candidate data, and a candidate index includes a candidate index value composed of N dimensions of key information and a corresponding candidate file offset; the candidate file offset corresponds to a candidate data. In this case, the N dimensions of key information may include at least one of the following: candidate data related identifiers, object related identifiers, and candidate data content keywords.

[0056] For example, the candidate index can be constructed based on multiple candidate data points located within the same preset time window and possessing the same candidate index value. Correspondingly, a candidate index includes a candidate index value composed of N dimensions of key information, and its corresponding multiple candidate file offsets; each candidate file offset corresponds to one candidate data point. It should be understood that in this case, the candidate index value composed of the N dimensions of key information can be a common candidate index value possessed by multiple candidate data points. The N dimensions of key information can include at least one of the following: candidate window end timestamp, candidate data related identifier, and candidate data content keywords.

[0057] The determination of the target index matching the retrieval request based on the candidate index values ​​contained in the candidate index includes at least one of the following:

[0058] If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes.

[0059] If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes;

[0060] If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index.

[0061] If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

[0062] To be specific:

[0063] The identifier related to the data to be retrieved may specifically include the ID of the log to be retrieved. As explained in the foregoing embodiments, the number of candidate indexes can be one or more, and the candidate index values ​​in each candidate index can be key information in N dimensions.

[0064] Accordingly, if the search request includes an identifier related to the data to be searched, determining the target index that matches the identifier related to the data to be searched based on the candidate data-related identifiers in the candidate indexes may include:

[0065] If the retrieval request includes the ID of the log to be retrieved, one or more target indexes are determined from one or more candidate indexes whose candidate log IDs match the ID of the log to be retrieved.

[0066] The identifier related to the object to be searched may specifically include: the ID of the user to be searched. When the search request includes an identifier related to the object to be searched, determining the target index that matches the identifier related to the object to be searched, based on the object-related identifiers in the candidate indexes, may include:

[0067] If the search request contains the ID of the user to be searched, one or more target indexes are determined from one or more candidate indexes whose candidate user IDs match the ID of the user to be searched.

[0068] When the search request contains keywords related to the content to be searched, determining a target index that matches the keywords related to the content to be searched, based on the candidate data content keywords in the candidate index, may include:

[0069] If the search request contains keywords for the content to be searched, one or more target indexes are determined from one or more candidate indexes whose candidate index values ​​match the keywords for the content to be searched.

[0070] When the search request includes a time range to be searched, determining the target index within the time range to be searched based on the candidate window end timestamp in the candidate indexes may include:

[0071] If the search request includes a time range to be searched, the time range of candidate data corresponding to the one or more candidate indexes is determined based on the candidate window termination timestamps in the candidate index values ​​contained in the one or more candidate indexes respectively; from the one or more candidate indexes, one or more target indexes of the time range of the candidate data within the time range to be searched are determined.

[0072] The time range to be searched can be set according to the actual situation, such as 10:00-12:00; or it can be a specific date, such as today, yesterday or the day before yesterday; or it can be 1 week or 1 month, etc., which will not be exhaustively listed here.

[0073] The step of determining the time range of candidate data corresponding to one or more candidate indices based on the candidate window termination timestamps in the candidate index values ​​contained in one or more candidate indices may specifically include:

[0074] Based on the candidate window termination timestamp in the candidate index values ​​contained in the candidate index, and the duration of the preset time window, the candidate window start timestamp is determined; the range between the candidate window start timestamp and the candidate window termination timestamp is taken as the time range of the candidate data corresponding to the candidate index.

[0075] It should also be noted that the above situations can be used in combination, for example:

[0076] If the search request includes identifiers related to the data to be searched and identifiers related to the object to be searched, at least one index to be filtered is determined based on the candidate data-related identifiers in the candidate indexes; and one or more target indexes are determined based on the object-related identifiers in the at least one index to be filtered, which match the object-related identifiers to be searched.

[0077] If the search request includes a search time range and search content keywords, at least one filterable index within the search time range is determined based on the candidate window termination timestamp in the candidate index; and one or more target indexes matching the search content keywords are determined based on the candidate data related identifiers contained in the at least one filterable index.

[0078] It should be understood that the above is only a partial example of the combination of the two situations mentioned above, and does not mean that the above situations can only be combined in pairs. In actual processing, three or four situations can also be used in combination, but this is not an exhaustive list.

[0079] As can be seen, by adopting the above scheme, the matching target index value can be determined from the candidate index values ​​based on the different situations of the search keywords contained in the search request. In this way, the target index value can be selected more accurately from a large number of candidate index values ​​stored locally, thereby obtaining the target data more accurately and ensuring the overall processing efficiency and accuracy.

[0080] In one implementation, obtaining target data from locally stored candidate data based on the target file offset contained in the target index includes:

[0081] Based on the target file offset contained in the target index, determine the location where the target data is stored in the local target file;

[0082] Based on the location where the target data is stored in the target file, the target data is obtained from the candidate data stored in the target file.

[0083] The target file offset may specifically include: the identifier of the target file, and the offset of the target data in the target file relative to the beginning position of the target file; and / or, the target file offset may specifically include: the identifier of the target file, and the offset of the target data in the target file relative to the end position of the target file.

[0084] It should be understood that the number of target files can be one or more; the target files are one or more of a plurality of candidate files stored locally. The number of target data can also be one or more, and further, the target data can specifically be one or more target data contained in one or more target files respectively.

[0085] The target file can specifically be a file used locally to store one or more candidate data. Furthermore, the identifier of the target file can be a unique identifier locally, for example, an identifier for a target file can be represented as: iNode = A1. The storage capacity of the target file can be preset, for example, a target file can store 300 bytes.

[0086] The offset of the target data relative to the beginning of the target file can be represented by a specific numerical value.

[0087] When the target index corresponds to a single target data item, the target file offset can be one. For example, see... Figure 2 Assuming the target file offset is represented as "iNode = A1 + 100", then the position of the target data a1 is the data at the offset of 100 relative to the beginning of the target file A1.

[0088] When the target index corresponds to multiple target data in the same target file, the target file offset can be multiple. For example, assuming the target file offset is represented as "iNode = A1 + 1 - 300", it means that the data in target file A1 with an offset of 1 relative to the beginning of target file A1 and an offset of 300 relative to the beginning of target file A1 are all the target data.

[0089] In this way, the local storage location of the target data can be determined based on the target file offset, and then the target data can be retrieved. This avoids the problem of high network overhead caused by retrieving the corresponding target data from the server over the network, thus ensuring retrieval efficiency and accuracy.

[0090] Before executing the solution provided in the above embodiments, it is necessary to pre-build the aforementioned candidate index. This will be explained in detail below:

[0091] In one implementation, it may include:

[0092] Based on the locally stored candidate data, the candidate index value is generated;

[0093] The candidate index is generated based on the candidate file offset corresponding to the candidate data and the candidate index value.

[0094] The candidate file offset corresponding to the candidate data can be obtained by determining the candidate file offset corresponding to the candidate data based on the location where the candidate data is stored in the local candidate file.

[0095] As can be seen, by adopting the above scheme, candidate indexes can be pre-built for locally stored candidate data, thus avoiding the low efficiency problem caused by generating candidate indexes for a large number of candidate data from other servers in the existing technology.

[0096] The step of generating the candidate index value based on the locally stored candidate data includes:

[0097] Key information is obtained by filtering from the locally stored candidate data; the key information is used as the candidate index of the candidate data.

[0098] In other words, a corresponding candidate index value can be generated for each candidate data.

[0099] Specifically, the step of filtering key information from locally stored candidate data can be to filter key information in N dimensions from locally stored candidate data.

[0100] Specific processing methods may include:

[0101] Based on regular expressions, key information across the N dimensions is extracted from the candidate data. The regular expressions can be set according to actual circumstances and are not limited here. For example, if user A queries the weather at time B, the corresponding candidate data (i.e., the corresponding candidate logs) will show that user A sent a query request containing weather information. If the regular expression includes user ID, request ID, and search keywords, then based on this regular expression, key information across the dimensions of user A, request ID, and search keywords can be extracted from the logs generated by the aforementioned query request.

[0102] Alternatively, based on filtering conditions, key information for the N dimensions can be obtained from the candidate data. The filtering conditions can be set according to actual circumstances; for example, they can be used to filter out content keywords containing user IDs, data IDs, preset types, etc., from the candidate data, and are not limited here.

[0103] The step of generating the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value includes: determining the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file; and generating the candidate index of the candidate data based on the candidate file offset and the candidate index value.

[0104] For example, assuming the candidate data is stored in the local candidate file at an offset of offset-C1 relative to the beginning of the candidate file, the candidate file offset corresponding to the candidate data is represented as: the candidate file identifier (e.g., inode A1) and the offset offset-C1. Assuming the candidate index value of the candidate data is represented as index1, then the candidate index of the candidate data can be represented as index1: inode A1, offset-C1.

[0105] In this way, corresponding key information can be generated for each candidate data point, and a candidate index value can be constructed based on the key information. This allows for richer dimensions in subsequent candidate data retrieval, ensuring the accuracy of the retrieval results.

[0106] In one implementation, generating the candidate index value based on the locally stored candidate data includes:

[0107] Key information is obtained by filtering from the locally stored candidate data;

[0108] The candidate data with the same key information within a preset time window are aggregated to obtain a candidate data group;

[0109] Based on the same key information and the window termination timestamp of the preset time window, the candidate index value of the candidate data group is determined.

[0110] The step of filtering key information from the locally stored candidate data can specifically involve filtering key information in N dimensions from the locally stored candidate data. The method for obtaining the key information in N dimensions is the same as in the previous embodiment and will not be repeated here.

[0111] The preset time window can be called the convergence window. The length of the preset time window can be set according to the actual situation. For example, it can be 3 seconds, 2 seconds, or 5 seconds, or it can be longer or shorter. This embodiment does not exhaustively list them.

[0112] The step of aggregating candidate data with the same key information within a preset time window to obtain a candidate data group may specifically include:

[0113] Extract multiple candidate data contained within the preset time window;

[0114] Based on the key information of N dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain a candidate data group with the same key information.

[0115] The same key information can be one or more key information from the N dimensions.

[0116] Wherein, the step of aggregating the multiple candidate data based on the key information of N dimensions corresponding to the multiple candidate data to obtain a candidate data group with the same key information can be at least one of the following:

[0117] Based on the content keywords in the key information of N dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same content keywords;

[0118] Based on the object-related information in the N dimensions of key information corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same object-related information.

[0119] For example, there may be multiple candidate data points containing the keyword "weather" within the preset time window. These multiple candidate data points can be grouped together as candidate data points sharing the same keyword "weather". For example, if there are multiple candidate data points for user A within the preset time window, all candidate data points for user A can be grouped together as a single candidate data point.

[0120] It should be understood that if multiple candidate data are aggregated based on the aforementioned key information, some candidate data may appear in multiple candidate data groups. For example, if user A can search for the keyword "weather," the candidate data generated by user A's weather search may be simultaneously categorized into one candidate data group with "weather" as the content keyword and another candidate data group for user A.

[0121] Determining the candidate index value of the candidate data group based on the same key information and the window termination timestamp of the preset time window may include:

[0122] The window termination timestamp of the preset time window is used as a new key information, and the new key information and the same key information are used together as the candidate index value of the candidate data group.

[0123] In other words, the window termination timestamp of the preset time window is added to the candidate data group based on the same key information, and together they serve as the candidate index value of the candidate data group.

[0124] As can be seen, by adopting the above scheme and aggregating candidate data within a preset time window, one candidate index can correspond to one or more candidate file offsets. This further reduces the storage space occupied by storing the index.

[0125] In one implementation, generating the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value includes:

[0126] Based on the location of the candidate data contained in the candidate data group in the local candidate file, determine the offset of the candidate file corresponding to the candidate data group;

[0127] The candidate index of the candidate data group is generated based on the candidate index value of the candidate data group and the candidate file offset corresponding to the candidate data group.

[0128] The above candidate data set is any one of all candidate data sets. In other words, the above scheme can be used to process each candidate data set, but it will not be described in detail.

[0129] Determining the offset of the candidate file corresponding to the candidate data group based on the location of the candidate data contained in the candidate data group in the local candidate file may include:

[0130] Based on the location of all candidate data contained in the candidate data group in the local candidate file, determine the candidate file offset corresponding to each of the candidate data in the candidate data group.

[0131] Generating the candidate index of the candidate data group based on the candidate index value of the candidate data group and the candidate file offset corresponding to the candidate data group may include:

[0132] Based on the candidate index value of the candidate data group and the candidate file offsets corresponding to all candidate data in the candidate data group, the candidate index corresponding to the candidate data group is generated.

[0133] In other words, the candidate index corresponding to the candidate data group may contain a set of candidate index values, as well as the candidate file offsets of all candidate data that are commonly corresponding to the candidate index values.

[0134] As can be seen, by adopting the above scheme, candidate data within a preset time window can be aggregated, allowing one candidate index to correspond to one or more candidate file offsets. This further reduces the storage space occupied by the index. Furthermore, since a corresponding candidate index value is generated for each group of candidate data, the candidate file offsets of multiple candidate data matching the search request can be obtained at once during the retrieval process, and the corresponding candidate data can be used as the target data, thereby further improving retrieval efficiency.

[0135] The second aspect of this disclosure also provides a data retrieval system, such as... Figure 3 As shown, it includes: a data retrieval device 301 and a first device 302; wherein,

[0136] The data retrieval device 301 is configured to receive a retrieval request from a first device; determine a target index matching the retrieval request based on the candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data; obtain target data from the locally stored candidate data based on the target file offset contained in the target index; and feed back the target data as a retrieval result to the first device.

[0137] The first device 302 is used to send a search request to the data retrieval device and receive the search results fed back by the data retrieval device.

[0138] The data retrieval device 301 can be one or more. For example, the data retrieval system can include K data retrieval devices (K is an integer greater than or equal to 2), and the K data retrieval devices are respectively set in K servers; each data retrieval device can specifically serve (or act as a service agent) for the log-agent on its server.

[0139] The first device can be a user-side device, and a log-query code or service can be set in the first device; the first device sends a retrieval request to the data retrieval device through the log-query code or service, and receives the retrieval results fed back by the data retrieval device.

[0140] The first device 302 is used to aggregate the target data contained in the search results fed back by the data retrieval device to obtain aggregated search results.

[0141] Specifically, the search results received by the first device from the data retrieval device through the log-query code or service may include one or more target data. Therefore, the first device aggregates the received target data (i.e., target logs) through the log-query code or service to finally obtain the aggregated search results. For example, if the first device obtains a total of 1000 target data entries from the data retrieval device through the log-query code or service, aggregates the 1000 target data entries to obtain 100, and uses these 100 target data entries as the final aggregated search results.

[0142] Specifically, the first device may aggregate data based on a data retrieval device and / or an IP (Internet Protocol) address through the log-query code or service.

[0143] The data retrieval device 301 is configured to perform at least one of the following:

[0144] If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes.

[0145] If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes;

[0146] If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index.

[0147] If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

[0148] The data retrieval device 301 is used to determine the location where the target data is stored in a local target file based on the target file offset contained in the target index; and to retrieve the target data from the candidate data stored in the target file based on the location where the target data is stored in the target file.

[0149] The data retrieval device 301 is used to generate candidate index values ​​based on the locally stored candidate data; and to generate candidate indexes based on the candidate file offsets corresponding to the candidate data and the candidate index values.

[0150] The data retrieval device 301 is used to filter key information from the locally stored candidate data; use the key information as the candidate index value of the candidate data; determine the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file; and generate the candidate index of the candidate data based on the candidate file offset and the candidate index value.

[0151] The data retrieval device 301 is used to filter key information from the locally stored candidate data; aggregate the candidate data with the same key information within a preset time window to obtain a candidate data group; and determine the candidate index value of the candidate data group based on the same key information and the window termination timestamp of the preset time window.

[0152] The data retrieval device 301 is used to determine the offset of the candidate file corresponding to the candidate data group based on the location of the candidate data contained in the candidate data group in the local candidate file; and to generate the candidate index of the candidate data group based on the candidate index value of the candidate data group and the offset of the candidate file corresponding to the candidate data group.

[0153] In related technologies, the ELK (Elasticsearch, Logstash, Kibana) cluster architecture is as follows: Figure 4 As shown, in the ELK cluster, each application server in the application service cluster collects logs through its FileBeat (file log collector); the LogStash service cluster (which may contain...) Figure 4 Multiple "LogStash" instances, each acting as a LogStash (log storage) server, filter logs and forward them to the Elasticsearch (ES) service cluster (which may contain...). Figure 4 Multiple Elasticsearch (ES) clusters (each of which can be an ES server) are used in the ELK cluster architecture. The ES service cluster is the core of the ELK cluster architecture. The ES service cluster receives logs sent from the Logstash service cluster, stores them, and creates indexes, which can then be used by Kibana to search and retrieve data. Figure 4Kibana, in this context, is an open-source, browser-based Elasticsearch analytics and search dashboard. It's a web project accessible via a browser after deployment. The Elasticsearch service cluster is responsible for collecting logs, building indexes, and receiving search queries. In practice, a large number of log entries are generated per minute or second. The Elasticsearch service cluster continuously receives new logs from the log server, resulting in significant network overhead. Furthermore, the Elasticsearch service cluster creates an index for each log entry, requiring a large amount of index data to be stored. These network overheads and large index data volumes delay index building, leading to inefficient index creation and consequently, lower search efficiency.

[0154] Finally, combining Figure 5 and Figure 6 Taking the data retrieval device as a log-agent set up in K servers, the K servers as log servers, the code or service that generates retrieval requests and obtains retrieval results by the first device as a log query, and candidate data as candidate logs and target data as target logs as an example, the processing in the system provided in this embodiment will be explained as follows:

[0155] See Figure 5 Let's take any one of the K servers as an example for explanation. Figure 5 In this context, any one of the K servers can be represented as a log server. Figure 5 The log-agent in this context refers to the data retrieval device set up in the log server.

[0156] The log-agent (i.e., the data retrieval device) determines the candidate index value and candidate file offset of the candidate logs based on the candidate logs stored locally on the log server. The candidate index value and the candidate file offset are then used as the candidate index of the candidate log.

[0157] For example, taking user A querying weather information at a certain time B as an example, the query request ID (i.e., log-ID), user ID (e.g., user A), and content keywords (e.g., the keyword is weather information) in the candidate log are filtered out from the candidate log as the above candidate index values; and a candidate index is built based on the above candidate index values.

[0158] During the creation of the candidate index, since the candidate logs are stored locally on the log server, the log-agent (i.e., the data retrieval device) does not need to save the original logs, but only saves the candidate index value (e.g., 8 bytes) and the candidate file offset (e.g., 8 bytes) as the candidate index of the candidate logs. This can greatly reduce storage costs.

[0159] The log-agent (i.e., the data retrieval device) aggregates candidate logs with the same key information based on a preset time window (e.g., a convergence window) to obtain a candidate log group. Based on the same key information and the window's end timestamp, a candidate index value for the candidate log group is determined. The candidate file offset of the candidate log group can refer to the start and / or end offsets corresponding to all candidate logs within the group. Furthermore, based on the candidate index value and the corresponding candidate file offsets, a candidate index for the candidate log group is generated. This candidate index can be in (Key-Value) form, where the key can include the same key information and the window's end timestamp, and the value can be a unique identifier (e.g., an inode) of the candidate log file containing the candidate logs, as well as the start / end offsets of the candidate logs within that file.

[0160] For example, see Figure 5 The log-agent (i.e., the data retrieval device) first performs the collection process, specifically: filtering candidate logs to obtain candidate index values; the log-agent (i.e., the data retrieval device) then performs the indexing (i.e., index generation) process, specifically: aggregating candidate logs with the same key information based on a preset time window to obtain a candidate log group; and generating a candidate index value (e.g., based on the same key information and the window termination timestamp of the preset time window) for the candidate log group. Figure 5 In the candidate log group, each candidate log file corresponds to a candidate file offset (e.g., represented as offset), and the candidate index of the candidate log group is finally constructed. The candidate index is as follows: Figure 5 As shown in the diagram, each idx1 corresponds to the offsets of offset1, offset2 to offset1+N candidate files.

[0161] When the log-agent (i.e. the data retrieval device) stores the candidate index, it can split the currently stored candidate index by hour and clean (or query) it according to a preset time interval.

[0162] The generation, writing, and reading of the aforementioned candidate indexes can be based on... Figure 5 The kv storage engine implementation is shown. Figure 5 In the kv storage engine shown, each candidate index can correspond to a db (database) in leveldb (i.e., kv database).

[0163] Furthermore, the log-agent (i.e., the data retrieval device) performs retrieval processing, that is... Figure 5 Specifically, the retrieval section shown in the figure: after receiving a retrieval request, the log-agent (i.e., the data retrieval device) can read the target log matching the retrieval request from the kv storage engine as the retrieval result.

[0164] Combination Figure 6 The code or service of the first device (i.e. Figure 6 The process of initiating retrieval requests to multiple log-agents (i.e., the data retrieval devices) and obtaining retrieval results in the context of log-query is illustrated by example. Assuming K=2, specifically:

[0165] The code or service of the first device (i.e. Figure 6 The log query in the middle sends queries to two log servers simultaneously. Figure 6 The log-agent (i.e., the data retrieval device) in the log sends a retrieval request;

[0166] The log-agent (i.e., the data retrieval device) of the two log servers retrieves the target log from its respective log server based on the retrieval request, and uses the target log as code or service to return the retrieval results to the first device. Figure 6 (log-query in the middle);

[0167] The code or service of the first device (i.e. Figure 6 After receiving the search results from the log-agent (i.e., the data retrieval device) of the two log servers, the log-query in the middle aggregates the results according to the module and / or IP address to obtain the aggregated search results, and finally feeds them back to the caller.

[0168] A third aspect of this disclosure provides a data retrieval device, such as... Figure 7 As shown, it includes:

[0169] The receiving module 701 is used to receive the retrieval request sent by the first device;

[0170] The index matching module 702 is used to determine the target index that matches the retrieval request based on the candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data;

[0171] The retrieval module 703 is used to retrieve target data from the locally stored candidate data based on the target file offset contained in the target index;

[0172] The sending module 704 is used to send the target data back to the first device as a retrieval result.

[0173] The index matching module is configured to perform at least one of the following:

[0174] If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes.

[0175] If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes;

[0176] If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index.

[0177] If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

[0178] The retrieval module is used to determine the location where the target data is stored in the local target file based on the target file offset contained in the target index; and to obtain the target data from the candidate data stored in the target file based on the location where the target data is stored in the target file.

[0179] like Figure 8 As shown, the device further includes:

[0180] The initial processing module 801 is used to generate the candidate index value based on the locally stored candidate data;

[0181] The index generation module 802 is used to generate the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value.

[0182] The initial processing module 801 is used to filter out key information from the locally stored candidate data and use the key information as the candidate index value of the candidate data.

[0183] The index generation module 802 is used to determine the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file; and to generate the candidate index of the candidate data based on the candidate file offset and the candidate index value.

[0184] The initial processing module 801 is used to filter out key information from the locally stored candidate data; aggregate the candidate data with the same key information within a preset time window to obtain a candidate data group; and determine the candidate index value of the candidate data group based on the same key information and the window termination timestamp of the preset time window.

[0185] The index generation module 802 is used to determine the offset of the candidate file corresponding to the candidate data group based on the location of the candidate data contained in the candidate data group in the local candidate file; and to generate the candidate index of the candidate data group based on the candidate index value of the candidate data group and the offset of the candidate file corresponding to the candidate data group.

[0186] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0190] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0191] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the various methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the various methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the various methods described above by any other suitable means (e.g., by means of firmware).

[0192] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0196] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0197] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0198] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0199] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data retrieval method, comprising: Receive the search request sent by the first device; Based on the candidate index values ​​contained in the candidate index, a target index matching the retrieval request is determined; wherein, the candidate index is constructed based on locally stored candidate data; Based on the target file offset contained in the target index, the target data is obtained from the locally stored candidate data; The target data is fed back to the first device as a search result; The method further includes: generating the candidate index value based on the locally stored candidate data; and generating the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value. The step of generating the candidate index value based on the locally stored candidate data includes: filtering out key information from the locally stored candidate data; aggregating the candidate data with the same key information within a preset time window to obtain a candidate data group; and determining the candidate index value of the candidate data group based on the same key information and the window termination timestamp of the preset time window. The step of aggregating candidate data with the same key information within a preset time window to obtain candidate data groups includes: extracting multiple candidate data contained within the preset time window; aggregating the multiple candidate data based on content keywords in the key information of multiple dimensions corresponding to the multiple candidate data to obtain multiple candidate data groups with the same content keywords; and aggregating the multiple candidate data based on object-related information in the key information of multiple dimensions corresponding to the multiple candidate data to obtain multiple candidate data groups with the same object-related information.

2. The method according to claim 1, wherein, The determination of the target index matching the retrieval request based on the candidate index values ​​contained in the candidate index includes at least one of the following: If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes. If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes; If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index. If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

3. The method according to claim 1, wherein, The step of obtaining target data from locally stored candidate data based on the target file offset contained in the target index includes: Based on the target file offset contained in the target index, determine the location where the target data is stored in the local target file; Based on the location where the target data is stored in the target file, the target data is obtained from the candidate data stored in the target file.

4. The method according to claim 1, wherein, The step of generating the candidate index value based on the locally stored candidate data includes: filtering out key information from the locally stored candidate data; and using the key information as the candidate index value of the candidate data. The step of generating the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value includes: determining the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file; and generating the candidate index of the candidate data based on the candidate file offset and the candidate index value.

5. The method according to claim 1, wherein, The step of generating the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value includes: Based on the location of the candidate data contained in the candidate data group in the local candidate file, determine the offset of the candidate file corresponding to the candidate data group; The candidate index of the candidate data group is generated based on the candidate index value of the candidate data group and the candidate file offset corresponding to the candidate data group.

6. A data retrieval system, comprising: Data retrieval device, first device; wherein, The data retrieval device is configured to receive a retrieval request from a first device; determine a target index matching the retrieval request based on candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data; obtain target data from the locally stored candidate data based on the target file offset contained in the target index; and feed back the target data as a retrieval result to the first device. The first device is configured to send a search request to the data retrieval device and receive the search results fed back by the data retrieval device; The data retrieval device is used to generate candidate index values ​​based on the locally stored candidate data; and to generate the candidate index based on the candidate file offsets corresponding to the candidate data and the candidate index values. The data retrieval device is used to filter key information from locally stored candidate data; aggregate candidate data with the same key information within a preset time window to obtain a candidate data group; and determine the candidate index value of the candidate data group based on the same key information and the window termination timestamp of the preset time window. The data retrieval device is used to extract multiple candidate data contained within the preset time window; based on the content keywords in the key information of multiple dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same content keywords; based on the object-related information in the key information of multiple dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same object-related information.

7. The system according to claim 6, wherein, The first device is used to aggregate the target data contained in the search results fed back by the data retrieval device to obtain aggregated search results.

8. The system according to claim 6, wherein, The data retrieval device is configured to perform at least one of the following: If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes. If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes; If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index. If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

9. The system according to claim 6, wherein, The data retrieval device is used to determine the location where the target data is stored in the local target file based on the target file offset contained in the target index; Based on the location where the target data is stored in the target file, the target data is obtained from the candidate data stored in the target file.

10. The system according to claim 6, wherein, The data retrieval device is used to filter key information from the locally stored candidate data; use the key information as the candidate index value of the candidate data; and determine the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file. Based on the candidate file offset and the candidate index value, the candidate index of the candidate data is generated.

11. The system according to claim 6, wherein, The data retrieval device is used to determine the offset of the candidate file corresponding to the candidate data group based on the location of the candidate data contained in the candidate data group in the local candidate file. The candidate index of the candidate data group is generated based on the candidate index value of the candidate data group and the candidate file offset corresponding to the candidate data group.

12. A data retrieval device, comprising: The receiving module is used to receive retrieval requests sent by the first device; An index matching module is used to determine a target index that matches the retrieval request based on the candidate index values ​​contained in the candidate index; wherein the candidate index is constructed based on locally stored candidate data; The retrieval module is used to retrieve target data from the locally stored candidate data based on the target file offset contained in the target index; The sending module is used to send the target data back to the first device as a search result; The device further includes: an initial processing module, used to generate the candidate index value based on the locally stored candidate data; and an index generation module, used to generate the candidate index based on the candidate file offset corresponding to the candidate data and the candidate index value. The initial processing module is used to filter out key information from the locally stored candidate data; aggregate the candidate data with the same key information within a preset time window to obtain a candidate data group; and determine the candidate index value of the candidate data group based on the same key information and the window end timestamp of the preset time window. The initial processing module is used to extract multiple candidate data contained within the preset time window; based on the content keywords in the key information of multiple dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same content keywords; based on the object-related information in the key information of multiple dimensions corresponding to the multiple candidate data, the multiple candidate data are aggregated to obtain multiple candidate data groups with the same object-related information.

13. The apparatus according to claim 12, wherein, The index matching module is configured to perform at least one of the following: If the search request contains an identifier related to the data to be searched, a target index that matches the identifier related to the data to be searched is determined based on the candidate data-related identifiers in the candidate indexes. If the search request contains an identifier related to the object to be searched, a target index that matches the identifier related to the object to be searched is determined based on the object-related identifier in the candidate indexes; If the search request contains keywords for the content to be searched, a target index that matches the keywords for the content to be searched is determined based on the candidate data content keywords in the candidate index. If the search request includes a time range to be searched, the target index within the time range to be searched is determined based on the candidate window termination timestamp in the candidate index.

14. The apparatus according to claim 12, wherein, The retrieval module is used to determine the location where the target data is stored in the local target file based on the target file offset contained in the target index; Based on the location where the target data is stored in the target file, the target data is obtained from the candidate data stored in the target file.

15. The apparatus according to claim 12, wherein, The initial processing module is used to filter key information from the locally stored candidate data and use the key information as the candidate index value of the candidate data. The index generation module is used to determine the candidate file offset of the candidate data based on the location where the candidate data is stored in the local candidate file. Based on the candidate file offset and the candidate index value, the candidate index of the candidate data is generated.

16. The apparatus according to claim 12, wherein, The index generation module is used to determine the offset of the candidate file corresponding to the candidate data group based on the location of the candidate data contained in the candidate data group in the local candidate file. The candidate index of the candidate data group is generated based on the candidate index value of the candidate data group and the candidate file offset corresponding to the candidate data group.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Citation Information

Patent Citations

  • Log searching method and device

    CN106649627A