Log query method and device, medium, electronic equipment and program product

The log storage file is indexed through time index files and data segment index files, which solves the problem of high log retrieval cost in the existing technology, realizes low-cost and efficient log query performance, and meets the diverse log retrieval needs.

CN119003458APending Publication Date: 2024-11-22BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411216796.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the scenario of massive log data, existing log search engines improve retrieval performance by building inverted indexes, but it leads to high storage costs and complex index structure, making it difficult to meet the needs of low-cost and efficient log retrieval.

Method used

The time index file and data segment index file are used to index the log storage file through the time dimension and data segment dimension, reduce the indexing requirement for log content, reduce storage costs, and speed up the log query process through parallel query and metadata filtering.

Benefits of technology

It significantly reduces the storage cost of log indexes, improves log query performance, meets more log retrieval usage scenarios, and realizes low-cost and efficient log query.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003458A_ABST
    Figure CN119003458A_ABST
Patent Text Reader

Abstract

The invention relates to a log query method and device, a medium, electronic equipment and a program product. The log query performance can be improved. The method comprises the following steps: receiving a log query request, wherein the log query request is used for requesting to query target log content within a target time range; respectively searching a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on logs, and searching a target query range corresponding to the target time index from the target log storage file; constructing a query plan according to the target query range; and according to the query plan, controlling the working node to concurrently search a data segment index matched with a data block specified by the allocated query subtask from a data segment index file corresponding to the target log storage file, and querying target log content based on a target data block corresponding to the data segment index.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of log query, and in particular, to a log query method, device, medium, electronic device and program product. Background Art

[0002] Currently, in the case of massive log data, log search engines usually speed up the retrieval process by building indexes. The currently very popular log search engines will segment and process the document content during the data writing process and build an inverted index, but this method requires more storage space to store index information. As the amount of data increases, the storage cost required for the index will also increase.

[0003] Therefore, how to provide a low-cost and efficient log search engine to significantly reduce the storage cost brought by indexing, and to improve the performance of log retrieval as much as possible on a low-cost basis to meet more log retrieval usage scenarios is a technical problem that urgently needs to be solved. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a log query method, comprising: receiving a log query request, the log query request being used to request a query for target log content within a target time range; in response to the log query request, searching for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, respectively, and searching for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; constructing a query plan according to the target query range, the query plan being used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; according to the query plan, controlling working nodes to search in parallel for data segment indexes matching the data blocks specified by the assigned query subtasks from a data segment index file corresponding to the target log storage file, and querying the target log content based on the target data blocks corresponding to the data segment indexes, the data segment index file being dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0006] In a second aspect, the present disclosure provides a log query device, comprising: a receiving module, used to receive a log query request, wherein the log query request is used to request a query for a target log content within a target time range; a log query module, used to, in response to the log query request, search for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, respectively, and search for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; a query plan is constructed according to the target query range, wherein the query plan is used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; according to the query plan, a working node is controlled to search in parallel from a data segment index file corresponding to the target log storage file for a data segment index matching the data block specified by the assigned query subtask, and query the target log content based on the target data block corresponding to the data segment index, wherein the data segment index file is dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0007] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any one of the methods described in the first aspect.

[0008] In a fourth aspect, the present disclosure provides an electronic device, including:

[0009] a storage device having a computer program stored thereon;

[0010] A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods in the first aspect.

[0011] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0012] By adopting the above technical solution, the index used in the log query process includes a time index file and a data segment index file. Since the time index file indexes the log storage file by time period from the time dimension, and the data segment index file indexes the log storage file by data segment from the data segment dimension, there is no need to index by each word in the log content and the document ID of the word as in the related art. Therefore, the storage cost of the log index is greatly reduced. In addition, in the log query process, the time index file and the data segment index file can be combined to perform log queries from the time dimension and the data segment dimension respectively. Among them, the target query range within the target time range requested by the log query request can be determined through the time index file, and the target query range can be further narrowed through the data segment index file, thereby speeding up the log query process, improving the performance of the log query, and satisfying more log query usage scenarios.

[0013] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0015] Figure 1 It is a schematic diagram of a log index structure according to the related art.

[0016] Figure 2 It is a schematic diagram of the structure of a time index file according to an embodiment of the present disclosure.

[0017] Figure 3 It is a schematic diagram of merging the generated time indexes according to an embodiment of the present disclosure.

[0018] Figure 4 It is a schematic diagram of the structure of a data segment index file according to an embodiment of the present disclosure.

[0019] Figure 5 is a schematic diagram of a shared prefix tree according to an embodiment of the present disclosure.

[0020] Figure 6 It is a schematic diagram of merging the generated data segment indexes according to an embodiment of the present disclosure.

[0021] Figure 7 The figure is a flow chart of a log query method according to an embodiment of the present disclosure.

[0022] Figure 8It is a schematic diagram of merging target time indexes according to an embodiment of the present disclosure.

[0023] Fig. 9 It is a schematic diagram of parsing a structured log according to an embodiment of the present disclosure.

[0024] Fig.10 It is a schematic diagram of the overall architecture of log query according to an embodiment of the present disclosure.

[0025] Fig.11 It is a schematic block diagram of a log query device according to an embodiment of the present disclosure.

[0026] Fig.12 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] Before describing in detail the log query method according to the embodiment of the present disclosure, the log index structure according to the embodiment of the present disclosure is first described.

[0034] Log indexing can speed up the log query process. In related technologies, the common Elasticsearch (ES) engine will segment and process the content of log documents and build an inverted index as the log index. An inverted index is a structure that uses words in the document content as an index and the document ID containing the word as a record. This structure usually needs to contain two parts of data, such as Figure 1 As shown in the figure, the first part of the data is the index table (TermDictionary), which consists of a series of words (Term) generated after the content in the document is segmented; the second part of the data is the record table (PostingsList), which contains information such as document ID. When querying logs, first find the relevant words in the index table, then obtain the corresponding record table, and then obtain detailed information in the log document based on the document ID and other information in the record table.

[0035] The above log index structure will bring additional storage costs. For example, under the general configuration of ES, the expansion ratio of index and original log data is about That is, assuming the original log data volume is 1 TB, the log index requires approximately The larger the amount of raw log data, the higher the log index storage cost will be.

[0036] According to the log index structure of the embodiment of the present disclosure, the log index structure includes a time index file and a data segment index file, which are used to index the original log storage file from the time dimension and the data segment dimension respectively. The time index file and the log storage file are in a one-to-one correspondence. The log storage file and the data segment index file can be a one-to-many relationship or a one-to-one relationship.

[0037] Log storage files are used to store logs and can be obtained in the following ways.

[0038] The log storage file may be a log storage file obtained according to a preset time granularity of log storage. Hereinafter, the preset time granularity of log storage is represented by a first time granularity. When generating the log storage file, the logs may be written into different log storage files according to the first time granularity.

[0039] The first time granularity may be 1 hour, 1 day, etc., which is not limited in the present disclosure.

[0040] For each log within the first time granularity, the log within the first time granularity can be written into one or more log storage files corresponding to the first time granularity according to the preset storage capacity of the single log storage file corresponding to the first time granularity and the amount of log data within the first time granularity. That is, if the amount of log data within the first time granularity is less than or equal to the preset storage capacity of the single log storage file corresponding to the first time granularity, the log within the first time granularity is written into a log storage file corresponding to the first time granularity; if the amount of log data within the first time granularity is greater than the preset storage capacity of the single log storage file corresponding to the first time granularity, the log within the first time granularity is respectively written into the multiple log storage files corresponding to the first time granularity. The preset storage capacity of the log storage file can be a preset number of log storage items or a preset storage space occupancy (for example, in bytes). The amount of log data within the first time granularity can be the total number of logs within the first time granularity or the amount of storage space occupied by the logs within the first time granularity (for example, in bytes).

[0041] Each first time granularity can be identified by a corresponding first time granularity identifier (DataTime). The log storage file corresponding to each first time granularity can be identified by the first time granularity identifier of the first time granularity plus the file identifier (hpathid) of the log storage file, that is, for example, by "[DataTime]-[hpathid]". For example, the log storage file can be identified by including "[DataTime]-[hpathid]" in the file name of the log storage file.

[0042] Taking the first time granularity of 1 hour as an example, for each first time granularity, logs belonging to different hours can be written into the log storage file corresponding to the corresponding hour. For example, the logs in the hour from 10:00 to 11:00 on June 18, 2024 are written into two log storage files, then the first time granularity identifier of this hour can be 2024061810, the file identifier of the first log storage file in the two log storage files can be 1, and the file identifier of the second log storage file can be 2, then the first log storage file can be identified by "2024061810-1", and the second log storage file can be identified by "2024061810-2".

[0043] The time index file includes multiple time indexes. The time index file and the time index are used to index the log storage file from the time dimension. The time index file and the time index included in the time index file can be obtained in the following way.

[0044] For each log storage file, according to the preset time granularity of the time index, the log of the time granularity can be obtained from the log storage file and parsed to obtain the time and storage address of the log of the time granularity, and then according to the storage address information of the log of the time granularity and the time identifier (TimeID) of the log of the time granularity determined based on the time granularity, a corresponding time index (TimeRecord) is generated, and the time index is sequentially written into the time index file to obtain a time index file for the log storage file, wherein the time index corresponds one-to-one with the log of the time granularity, and the time index file corresponds one-to-one with the log storage file. The preset time granularity of the time index is smaller than the preset time granularity of the log storage.

[0045] Hereinafter, the preset time granularity of the time index is represented by the second time granularity.

[0046] For each time index, the time index may include a storage address of the log corresponding to the time index in the log storage file and a time identifier of the log corresponding to the time index.

[0047] The time index can be a time sparse index. The time index is built based on the time in the log, and is intended to determine the log query range through the time index in the subsequent log query process.

[0048] The size of the second time granularity is configurable, for example, it can be 1 second, 1 minute or other sizes. The first time granularity can include multiple second time granularities.

[0049] For each time index, the storage address of the log corresponding to the time index in the log storage file may include the starting position (startPos) and ending position (endPos) of the log corresponding to the time index in the log storage file, or may include the offset information (offset) of the log corresponding to the time index in the log storage file and the length of the log corresponding to the time index. The offset information of the log corresponding to the time index in the log storage file refers to the offset of the log corresponding to the time index relative to the starting position of the log storage file.

[0050] For each second time granularity, the time identifier of the second time granularity is used to indicate the number of the second time granularity in the first time granularity to which the second time granularity corresponds. For example, if the first time granularity is 1 hour and the second time granularity is 1 minute, the time identifier of the second time granularity is used to indicate the number of minutes in the first time granularity to which the second time granularity corresponds, and the value range of the time identifier of the second time granularity can be 0 to 59.

[0051] In addition to storage address information and time identification information, the time index may also include other metadata information, such as more detailed log classification information for the log.

[0052] The time index file corresponding to the log storage file can be identified by the first time granularity identifier (DataTime) of the first time granularity corresponding to the log storage file plus the file identifier (hpathid) of the log storage file, that is, for example, by "[DataTime]-[hpathid]", so that the time index file and the log storage file can be made to correspond one to one, which is convenient for rapid positioning in the subsequent log query process. For example, the log index file can be identified by including "[DataTime]-[hpathid]" in the file name of the log index file.

[0053] The time index may be stored in a corresponding time index file in a variety of ways, for example, in an inverted index format.

[0054] Figure 2 It is a schematic diagram of the structure of a time index file according to an embodiment of the present disclosure. Figure 2 In the example, the first time granularity is 1 hour and the second time granularity is 1 minute. Figure 2 As shown, logs corresponding to different minutes are indexed with different time indexes.

[0055] Furthermore, due to the disorder of log time (e.g. Figure 2In the example, there is a log time disorder between the log in the nth minute and the log in the n+1th minute), etc., which may cause the time overlap and log range overlap between the generated time indexes. Therefore, it is necessary to further merge the generated time indexes to reduce the number of time indexes and further reduce the storage cost of the log index. Therefore, if the time identifiers of the two generated time indexes are the same and the interval of the log storage addresses corresponding to the two time indexes is less than the preset merge length threshold (MaxMergeLength), the two time indexes are merged into one time index and updated to the time index file. In this way, in the subsequent log query process, one or a small amount of read IOPS (Input / Output Operations PerSecond, number of I / O operations per second) can be used to load adjacent log ranges together, reducing the read IOPS pressure. If the time identifiers of the two generated time indexes are the same and the interval of the log storage addresses corresponding to the two time indexes is greater than or equal to the preset merge length threshold, the two time indexes are not merged. In this way, some unnecessary log data can be skipped as much as possible in the subsequent log query process, thereby improving the efficiency of log query.

[0056] The preset merge length threshold is configurable, for example, it may be 4 MB.

[0057] Figure 3 FIG. 1 is a schematic diagram of merging the generated time indexes according to an embodiment of the present disclosure. Figure 3 As shown, the generated time index 1 and time index 3 have the same time identifier, and the generated time index 2 and time index 4 have the same time identifier. Figure 3 The vertical rectangle in is used to represent the storage address of the log corresponding to the time index in the log storage file. Since the interval between the storage addresses indicated by the generated time index 1 and time index 3 is greater than the preset merge length threshold, the generated time index 1 is not merged with the time index 3. Since the interval between the storage addresses indicated by the generated time index 2 and time index 4 is less than the preset merge length threshold, the generated time index 2 and time index 4 are merged into time index 5. Therefore, the time index obtained after the merger includes time index 1, time index 3 and time index 5.

[0058] In actual production environments, the time of most logs is basically ordered, and the time deviation of adjacent logs is not large. Therefore, after the time index is generated for the first time, each second time granularity generally contains only one or a small number of time indexes. If the log time is seriously disordered, the logs within the same second time granularity are distributed in different locations of the log storage file. In this case, after the time index is generated for the first time, the same second time granularity may correspond to multiple time indexes, resulting in a larger storage space occupied by the log index. By continuously merging time indexes, the number of time indexes can be greatly reduced, reducing the storage cost of the log index.

[0059] The data segment index file and the data segment index (DataRecord) included in the data segment index file are used to index the log storage file from the dimension of the data segment. The data segment index file and the data segment index included in the data segment index file can be obtained in the following way.

[0060] For each log storage file, logs can be obtained from the log storage file and divided into multiple data segments according to a preset data segment segmentation granularity, wherein each data segment contains multiple logs; for each data segment, the data segment is parsed to obtain the key-value pair and the log storage address of each log; for each data segment, a data segment index is generated for the key of each log in the data segment, and is written into a corresponding data segment index file in a generated order, wherein the data segment index at least indicates the position of the corresponding data segment in the corresponding log storage file.

[0061] The log storage file can be split into multiple data segments according to a variety of splitting rules. For example, the data segment can be split according to the preset data segment size (SegmentSplitSize), or according to the preset number of log entries. The preset data segment size is configurable, for example, it can be 10MB or other sizes. The preset number of log entries is also configurable.

[0062] Each data segment includes multiple complete logs. In other words, each data segment needs to contain several complete logs. The size of each data segment can be different. Just try to split it evenly when splitting.

[0063] For each data segment, when generating the data segment index of the data segment, a corresponding data segment index can be generated for each key contained in the data segment, or a corresponding data segment index can be generated for some of the keys contained in the data segment according to actual needs. For example, if a data segment contains 3 keys, corresponding data segment indexes can be generated for these 3 keys. The selection of keys for generating data segment indexes can be preset, user-defined, etc., or some keys can be intelligently selected using a certain model through artificial intelligence (AI) to generate corresponding data segment indexes.

[0064] Data segment indexes with different keys can be stored and managed using different data segment index files. For example, assuming that a log storage file is divided into the first data segment, the second data segment, and the third data segment, data segment indexes are generated for the keys k1 and k2 contained in the first data segment, data segment indexes are generated for the keys k2 and k3 contained in the second data segment, and data segment indexes are generated for the keys k1 and k3 contained in the third data segment. Then, for the log storage file, there are three data segment index files finally obtained, namely, a data segment index file storing the data segment index of key k1, a data segment index file storing the data segment index of key k2, and a data segment index file storing the data segment index of key k3. In this way, data segment indexes with different keys are stored in different data segment index files.

[0065] The data segment index of each key may be stored in a corresponding data segment index file in a column storage format (eg, ORC format), for example.

[0066] For each data segment index file, the data segment index file can be identified by the first time granularity identifier (DataTime) of the first time granularity corresponding to the log storage file corresponding to the data segment index file, the file identifier (hpathid) of the log storage file corresponding to the data segment index file, and the key identifier (KeyID) of the key corresponding to the data segment index file, that is, for example, by "[KeyID]-[DataTime]-[hpathid]" to identify, so that a corresponding relationship can be established between the data segment index file and the log storage file and the key, which is convenient for rapid positioning in the subsequent log query process. For example, the data segment index file can be identified by including "[KeyID]-[DataTime]-[hpathid]" in the file name of the data segment index file.

[0067] The data segment index file can be a sparse index file.

[0068] The data segment index file can be generated in parallel with the time index file to improve the efficiency of log index generation. In the subsequent log query process, the time index file can be used to preliminarily determine the log range to be queried within the target time range requested by the log query request, and then the data segment index file can be used to further narrow the log query range and speed up the log query process.

[0069] Figure 4 Schematic diagram of the structure of the data segment index file according to the embodiment of the present disclosure. Figure 4 As shown, for each log storage file, the log storage file is first divided into several data segments, the logs in each data segment are parsed to obtain the key-value pair information contained in each log, and the data segment index of the key contained in the data segment is generated according to the key-value pair information. Then, the data segment indexes of the same keys in all data segments of the log storage file are stored in the same data segment index file to obtain the data segment index file corresponding to the log storage file.

[0070] For each data segment index, the data segment index may include a data header field (Header) for indicating the storage address information of the data segment corresponding to the data segment index in the log storage file. The storage address of the data segment corresponding to the data segment index in the log storage file may include the starting position and the ending position of the corresponding data segment in the log storage file, or may include the offset information (Offset) of the corresponding data segment in the log storage file and the length (Length) of the corresponding data segment. The offset information of the corresponding data segment in the log storage file refers to the offset of the corresponding data segment relative to the starting position of the log storage file.

[0071] The data header field may also include at least one of the following subfields: a data segment index length subfield (RecordLength), which is used to indicate the size of the data segment index; a data type subfield (LogType), which is used to indicate the data type of the key corresponding to the data segment index in the data segment corresponding to the data segment index, for example, a string type or a long type, etc.; a flag subfield (Flag), which is used to indicate which metadata is included in the data segment index, for example, in Flag, the relevant bits are used to indicate that the data segment index does not include the prefix tree field. The specific data type indicated by the data type subfield can be determined by index configuration and the like, and the index configuration can be set by the user or as a preset value, or can be set according to the specific system.

[0072] For each data segment index, the data segment index may also include at least one of the following fields: a minimum value field (minValue), a maximum value field (maxValue), a filter field (Filter), a prefix tree field (Tier), a suffix tree field (Suffix), a log entry field, or other metadata fields. In addition, the field sizes of these fields may be stored in the data header field.

[0073] The minimum value field is used to indicate the minimum value of the key corresponding to the data segment index in the data segment corresponding to the data segment index. The maximum value field is used to indicate the maximum value of the key corresponding to the data segment index in the data segment corresponding to the data segment index. In the subsequent log query process, some data segments that do not need to be queried can be quickly filtered out through the minimum value field and the maximum value field. For example, each log in a certain data segment contains the key "score", and in this data segment, the minimum value of the value corresponding to the key "score" is 60 and the maximum value is 80. When the queried data is not within the range defined by the minimum value and the maximum value (for example, the target log content requested by the log query request is "score>90"), there is no need to load the data segment, thereby saving read overhead and data parsing and filtering overhead, and improving the efficiency of log query.

[0074] When generating a data segment index, considering that the key-value (KV) in the log storage file is usually stored in text form, when obtaining the minimum and maximum values ​​of the key in the data segment, first convert the text value of the key into the data type indicated by the LogType subfield according to the information of the LogType subfield in the data header field, and then obtain the maximum and minimum values ​​of the key in the data segment. For example, in a data segment, a log has a key of "age" and its value is "30". According to the LogType subfield, it is determined that the data type should be a numeric type, so the text value "30" is converted into a numeric value "30", and then compared with the value of the "age" key in other logs in the data segment, and finally the minimum and maximum values ​​of the "age" key in the data segment are obtained.

[0075] The byte sizes occupied by the minimum value field and the maximum value field can be determined as follows:

[0076] (1) For fixed-length data, such as int32, long, or other types, the byte sizes of the minimum value field and the maximum value field can be determined based on the information in the LogType subfield;

[0077] (2) For data types of indefinite length, such as string types, the first several bytes (eg, 4 bytes) of the minimum value field and the maximum value field can be used to represent the actual byte size of the minimum value field and the maximum value field, respectively.

[0078] In addition, the minimum value field may include the byte size information of the minimum value field and the minimum value information of the key corresponding to the data segment index. The maximum value field may include the byte size information of the maximum value field and the maximum value information of the key corresponding to the data segment index.

[0079] The filter field is used to indicate all values ​​of the key corresponding to the data segment index in the data segment corresponding to the data segment index. The filter used in the filter field can be a Bloom filter. Through the Bloom filter, it is possible to quickly determine whether an element is in a certain set. In the process of generating the data segment index, all values ​​of the key corresponding to the data segment index in the data segment corresponding to the data segment index can be added to the Bloom filter. In the subsequent log query process, some data segments that do not need to be read can be filtered out through the Bloom filter, thereby improving the efficiency of the log query. For example, the values ​​corresponding to the "name" key of each log in a certain data segment are added to the Bloom filter. In the subsequent log query process, if the target log content requested by the log query request is "name:xiaoniing", the Bloom filter can be used to quickly determine whether "xiaoming" is likely to be in the data segment. If it is determined by the Bloom filter that it is impossible to exist in the data segment, the data segment is directly skipped.

[0080] The Bloom filter in the present disclosure can be implemented using the currently mature Bloom filters, or a Bloom filter can be developed independently according to actual needs.

[0081] The field size of the filter field is configurable, for example, 1KB. The length of the filter field can be represented by the first several bytes of the filter field, similar to the field size representation method of the indefinite length maximum value field / minimum value field.

[0082] The prefix tree field is used to indicate the shared prefix tree of the value of the key corresponding to the data segment index in the data segment corresponding to the data segment index. The maximum depth M of the shared prefix tree is configurable, for example, 3. The shared prefix tree is used to share the prefix of the value of the key corresponding to the data segment index, and the storage cost is low. In addition, by limiting the maximum depth of the shared prefix tree, the storage cost can be further reduced. When generating a shared prefix tree, if the length of a string is greater than M, only the first M characters of the string will be added to the shared prefix tree. In the subsequent log query process, the shared prefix tree can be used to filter data segments, such as Figure 5As shown in the shared prefix tree: Assuming M is 3, the string to be queried for logs is "bektest", the matching starts from the root node of the shared prefix tree. If the shared "bek" prefix is ​​found, and the matching process has reached the leaf node of the shared prefix tree, it means that the data segment may contain "bektest" and needs to be further matched and filtered in combination with other metadata; if the data to be queried for logs is "betest", and only the common prefix "be" is found, but the matching process has not reached the leaf node, it means that the data segment cannot contain the string and can be directly skipped in the log query process.

[0083] The larger the maximum depth M of the shared prefix tree, the higher the probability of data segment filtering during log query, and the storage cost of the log index is also relatively high. Assuming that the character set size of the key value corresponding to the data segment index is 26, the maximum size of the shared prefix tree is approximately 26^M bytes. When M is 3, the size of the shared prefix tree is approximately 18KB. In order to further reduce the storage cost of the log index, the shared prefix tree can be compressed by a compression algorithm such as Zstandard before being stored. Assuming the compression rate is 3, the size of the compressed shared prefix tree is approximately 6KB.

[0084] The prefix tree field may include field size information, compression type information, and shared prefix information of the prefix tree field, so as to facilitate data parsing and filtering in a subsequent log query process.

[0085] The suffix tree field is used to indicate the shared suffix tree of the value of the key corresponding to the data segment index in the data segment corresponding to the data segment index. Similar to the shared prefix tree, the shared suffix tree is used to store the common suffix of the value of the key corresponding to the data segment index. The maximum depth N of the shared suffix tree is configurable. When generating a shared suffix tree, if the length of a string exceeds N, only the last N characters of the string are added to the shared suffix tree. In the subsequent log query process, the shared suffix tree can also be used for filtering data segments, and the filtering process is similar to the shared prefix tree. In order to further reduce the storage cost of the log index, the shared suffix tree can be compressed before storage.

[0086] The suffix tree field may include field size information, compression type information, and shared suffix information of the suffix tree field, which facilitates data parsing and filtering in a subsequent log query process.

[0087] In the process of generating data segment indexes, there may be a situation where the size of the data segment corresponding to the data segment index is relatively small, for example, the log traffic is relatively low, resulting in a relatively small amount of logs generated within a certain period of time. In this scenario, after the data segment index is initially generated, the adjacent data segment indexes can be merged, that is, when the data volume of adjacent data segments is less than the preset data segment merging threshold, the data segment indexes corresponding to the adjacent data segments are aggregated into one data segment index and the data index file is updated to further reduce the storage space required for the data segment index.

[0088] Figure 6 FIG. 1 is a schematic diagram of merging the generated data segment indexes according to an embodiment of the present disclosure. Figure 6 As shown, the data segment corresponding to the generated data segment index 1 and the data segment corresponding to the generated data segment index 2 are adjacent to each other in the log storage file, and the sizes of the data segment corresponding to the generated data segment index 1 and the data segment corresponding to the generated data segment index 2 are both smaller than the preset data segment size. Then, the generated data segment index 1 and the generated data segment index 2 can be merged into a data segment index 3, wherein the minimum value in the data segment index 3 is the smallest of the minimum value in the generated data segment index 1 and the minimum value in the generated data segment index 2, the maximum value in the data segment index 3 is the largest of the maximum value in the generated data segment index 1 and the maximum value in the generated data segment index 2, and the Bloom filter 3 in the data segment index 3 is the merger of the Bloom filter 1 in the generated data segment index 1 and the Bloom filter 2 in the generated data segment index 2.

[0089] At this point, the log index for log query has been generated, and the generated log index includes a time index file and a data segment index file. Since the time index file indexes the log storage file by time period from the time dimension, and the data segment index file indexes the log storage file by data segment from the data segment dimension, it is not necessary to index by each word in the log content and the document ID of the word as in the related art, thus greatly reducing the storage cost of the log index.

[0090] The following example illustrates the storage cost of a log index according to an embodiment of the present disclosure.

[0091] The storage cost of the log index according to the embodiment of the present disclosure mainly includes two parts: the first part is the storage space occupied by the time index file; the second part is the storage space occupied by the data segment index file. The following is an estimate of the storage occupancy of these two parts. The following evaluation is based on the example of the first time granularity of 1 hour and the second time granularity of 1 minute.

[0092] In most cases, the time of adjacent logs is relatively close. After the generated time indexes are merged, in most cases, only one time index is obtained per minute. Considering that there may be some abnormal situations such as time disorder, it is estimated that 10 time indexes are obtained per minute (conservative estimate), and about 14,400 time indexes are generated every day. The start position, end position and other fields included in a single time index are all represented by 8B, and the time identifier is represented by 1B. The size of the entire time index is estimated to be 17B, so the time index generated in one day is less than 250KB. Compared with the size of the original log data, the size of the time index file is almost negligible.

[0093] The size of a single data segment index is not fixed. Each data segment index contains the data header field and various metadata information, which can be estimated as follows:

[0094] (1) The size of the RecordLength, Offset, and Length fields in the data header are all 4B, the size of the Flag is 2B, and the size of the LogType is 1B;

[0095] (2) The size of the minimum value field / maximum value field is related to LogType. For example, when LogType is a long data type, each minimum value field / maximum value field occupies 8B. For indefinite length types (such as string types), the size of the minimum value field / maximum value field of a single key is not fixed. Here, we estimate it to be about 100B, so the size of the minimum value field / maximum value field is about 200B;

[0096] (3) The filter size is also configurable, and is estimated here as 1KB;

[0097] (4) The size of the prefix tree field is not fixed either, but because it is a shared prefix and the maximum depth M is limited, and it is stored after compression, the size is relatively small. According to the character set size of 26, the maximum size of the prefix tree field is about 26^M bytes in the extreme case. Here, according to M = 3 and the compression rate = 3, the original prefix tree field occupies a maximum of 18KB, which can be reduced to about 6KB after compression.

[0098] (5) Similar to the evaluation process of the prefix tree field, the suffix tree field uses the same evaluation conditions and occupies approximately 6KB after compression.

[0099] Through the above evaluation, the size of the entire data segment index is estimated to be around 14KB. The number of keys in a data segment is estimated to be 10, and the storage size of the data segment index file corresponding to a data segment is about 140KB. The data segment size is configured as 10MB here, and the expansion ratio of the data segment index file is less than 0.02, which is much lower than the index expansion ratio of ES. The overall index storage cost can be less than 2% of the original log data. According to the estimate of 1TB of original log data per day, the log index storage space can be saved by about 1003.5GB per day, and the storage cost of the log index is greatly reduced.

[0100] Next, a log query method according to an embodiment of the present disclosure is described, which can be applied to various log query scenarios. The log query method can use the log index described above to perform log query. Figure 7 As shown, the log query method may include the following steps S71 to S74.

[0101] In step S71, a log query request is received, where the log query request is used to request a query for target log content within a target time range.

[0102] In step S72, in response to the log query request, the target time index in the target time index file and the target log storage file corresponding to the target time range are respectively searched from the time index file and the log storage file constructed based on the log, and the target query range corresponding to the target time index is searched from the target log storage file; wherein, the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file.

[0103] In order to ensure that all target time indexes within the target time range can be obtained, in step S72, it is necessary to fine-tune the start time and end time of the target time range before searching for the target time index, that is, round the start time of the target time range forward according to the second time granularity described above, and round the end time of the target time range backward according to the second time granularity described above. For example, assuming that the start time of the target time range is "2024-06-1815:10:53" and the end time is "2024-06-1818:10:53", and the second time granularity used when generating the time index is 1 minute, then it is necessary to round the start time of the target time range forward according to the minute to "2024-06-1815:10:00", and round the end time of the target time range backward according to the minute to "2024-06-1818:11:00". The purpose of such processing is that the time index is generated according to the second time granularity, and by rounding forward and rounding backward, it is ensured that all target time indexes within the target time range are obtained.

[0104] After the start time of the target time range is rounded forward according to the second time granularity and the end time of the target time range is rounded backward according to the second time granularity, the target time index and the target log storage file may be searched in the following manner.

[0105] The target log storage file can be determined based on the match between the file identifier of the log storage file and the target time range. For example, a file identifier list of the log storage file can be obtained, and the log storage file corresponding to the file identifier in the file identifier list that matches the target time range can be determined as the target log storage file. There can be many ways to obtain the file identifier list. For example, the file identifier list can be obtained by directly listing according to the first time granularity identifier in the file name of the log storage file; for another example, the file identifier list within each first time granularity can be indexed, and the file identifier list can be quickly determined based on the file identifier list index. In addition, after the file identifier list is initially obtained, the result can be cached to reduce the pressure on the storage system to list files.

[0106] After determining the target log storage files within the target time range, the target time index files corresponding to each of these target log storage files can be determined based on the one-to-one correspondence between the log storage files and the time index files, and then the time indexes in these target time index files can be obtained. After obtaining the time indexes in these target time index files, these time indexes can be cached to speed up the time index acquisition process and reduce the pressure on the storage system. In addition, in addition to determining the target time index file based on the target log storage file, the target time index file within the target time range can also be directly determined based on the matching of the time index file identifier and the target time range. The determination method is similar to the method of determining the target log storage file based on the file identifier described in the previous text.

[0107] After obtaining the time indexes in the target time index file, these time indexes can be traversed, and according to the time identifiers of the time indexes, the time indexes whose time identifiers match the target time range are determined as the target time indexes.

[0108] After the target time index is determined, the target query range corresponding to the target time index can be searched from the target log storage file. For example, the target query range is determined based on the storage address information in the target time index.

[0109] However, since the storage addresses corresponding to each target time index may overlap or be adjacent, after the target time index is determined, the target time index can be merged before searching the target query range corresponding to the target time index from the target log storage file. That is, if the interval between the storage addresses corresponding to two target time indexes is less than the preset merge length threshold, the two target time indexes are merged into one target time index, and then the target query range corresponding to the merged target time index is searched from the target log storage file. In this way, the log data within the target query range can be obtained from the storage system through a small amount of IOPS, thereby reducing the pressure on the storage system.

[0110] Figure 8 It is a schematic diagram of merging target time indexes according to an embodiment of the present disclosure. Figure 8 The vertical rectangle in is used to represent the storage address corresponding to the target time index. Figure 8 As shown, since the storage address indicated by the target time index 1 overlaps with the storage address indicated by the target time index 2, and the interval between the storage address indicated by the target time index 2 and the storage address indicated by the target time index 3 is less than the preset merge length threshold, the target time index 1, the target time index 2 and the target time index 3 are merged into a target time index 6, and since the storage address indicated by the target time index 4 overlaps with the storage address indicated by the target time index 5, the target time index 4 and the target time index 5 are merged into a target time index 7.

[0111] The preset merge length threshold can be dynamically adjusted according to the target time range in the log query request.

[0112] In step S73, a query plan is constructed according to the target query range, wherein the query plan is used to indicate query subtasks corresponding to the data blocks into which the target query range is segmented.

[0113] For example, constructing a query plan according to the target query range may be implemented in the following manner.

[0114] First, determine the number of data blocks to be accessed according to the target query range. Through the target time index, several relatively large target query ranges can usually be obtained. If all these target query ranges are scanned serially, the log query may take a long time and the user experience is relatively poor, so the target query range can be divided into multiple data blocks first. The size of the data block is configurable, for example, it can be 100MB. There can be multiple data block segmentation schemes. For example, the size of the data block can be determined according to factors such as the size of the target time range and the total amount of data in the target query range.

[0115] Then, according to the number of data blocks and the number of working nodes in an idle state, a plurality of query subtasks are generated, each query subtask is used to request access to a specified data block; then, according to the plurality of query subtasks, a query plan is generated.

[0116] In addition to the speed of log query, the formulation of query plan also needs to consider issues such as the fairness of log query requests, so as to avoid a single log query request occupying too many resources and affecting the execution of other log query requests. The query plan can be flexibly adjusted according to actual needs. For example, if the log query can only be executed in a single node and a single process, the log query speed can be accelerated by multi-threaded data block concurrency. Next, we will take the distributed system as an example to illustrate the generation of query plan.

[0117] Assume that the distributed system has N worker nodes, a single log query request generates L data blocks, and a single query subtask occupies at most P idle worker nodes. The distributed system can query the status of each worker node to determine whether there is an idle worker node. If there is an idle worker node, the data block is directly assigned to the idle worker node, and the assigned worker node is responsible for the actual log query process and returns the log query result. The number of worker nodes that can be occupied by a single log query request is in the range of [1, min (P, L)]. The data block that has not been assigned a worker node waits until there is an idle worker node or the log query request times out and is canceled. The worker node can be a process, thread or coroutine, which can be implemented according to the actual situation. Through parallel execution, data blocks can be queried in parallel according to a certain degree of concurrency, which speeds up the log query. In addition, in addition to considering the idle state of the worker node, the generation of the query plan can also consider factors such as the CPU and memory of the physical node where the worker node is located. In addition, the query plan described above is scheduled immediately as long as there is an idle worker node. In actual applications, scheduling can also be performed after waiting for a certain number of worker nodes to be idle.

[0118] In step S74, according to the query plan, the working nodes are controlled to search in parallel from the data segment index file corresponding to the target log storage file for a data segment index that matches the data block specified by the assigned query subtask, and the target log content is queried based on the target data block corresponding to the data segment index. The data segment index file is dynamically constructed based on the data segments into which the log in the log storage file is divided.

[0119] For example, when searching for a data segment index, you can search for the data segment index file corresponding to the target log storage file based on the correspondence between the log storage file and the data segment index file, and then search for the target data segment index file that matches the target log content requested by the log query request from these data segment index files, and then search for the data segment index that matches the data block specified by the assigned query subtask from the target data segment index file.

[0120] For example, for each data block, when loading the data segment index file corresponding to the target log storage file corresponding to the data block, it is only necessary to load the target data segment index file that matches the target log content requested by the log query request. For example, the file identifier of the target log storage file to which the data block belongs is 1, the target log content requested by the log query request is "score:80ANDname:xiaoming", and the target time range requested by the log query request is from 10:00 to 11:00 on June 18, 2024. Then, it is only necessary to load the target data segment index file with the following identifier, that is, "the first time granularity identifier matching the target time range + the file identifier of the target log storage file + the key identifier matching the target log content", for example, it is only necessary to load the data segment index file containing "score-2024061810-1" and "name-2024061810-1" in the file name, and other data segment index files corresponding to the target log storage file do not need to be loaded. After finding the target data segment index file, the data segment index matching the data block specified by the assigned query subtask can be found in the target data segment index file. For example, if the storage address corresponding to the data segment index 1 in the target data segment index file overlaps with the storage address of the data block specified by the assigned query subtask, the data segment index 1 is determined to be the data segment index matching the data block specified by the assigned query subtask. Then, by matching the data segment index matching the data block specified by the assigned query subtask with the target log content requested by the log query request, it is possible to estimate in advance whether the data block specified by the query subtask is likely to contain data matching the target log content requested by the log query request, that is, data block filtering can be achieved, for example, by matching the maximum value, minimum value, Bloom filter, shared prefix tree, shared suffix tree and other information in the data segment index with the target log content requested by the log query request. If it does not match, the corresponding data block is filtered out and no subsequent data loading, parsing, filtering and other processes are required, thereby reducing the pressure on the storage system and improving the efficiency of log query.

[0121] For example, querying the target log content based on the target data block corresponding to the data segment index may include: before each log data loading, first determine the data volume of the log data loaded this time, the data volume of the log data loaded this time being the minimum value between the expected value of the storage read traffic (BestIoSize) and the data volume of the target data block that has not loaded the data; and loading the log data according to the data volume of the log data loaded this time. The target data block here refers to the data block finally obtained through the data block filtering process described above. By controlling the data volume of the log data loaded at a single time, it is possible to avoid problems such as memory exhaustion caused by loading too much log data at a single time. The size of the expected value of the storage read traffic is configurable, for example, 4MB.

[0122] In addition, since a data block generally includes a header part and a data part, and the header part can also include some related metadata information, when loading the target data block, in addition to using the above-mentioned loading method, you can also load the header part first, perform metadata matching filtering first, and then load the data part.

[0123] In addition, since the original data in the data block is usually serialized and compressed to reduce storage costs and improve transmission efficiency, for example, serialization is performed through serialization technologies such as Protocol Buffers, and then compressed through compression algorithms (such as lz4), the loaded log data generally needs to perform decompression, deserialization and other processes, and then can be matched and filtered with the target log content requested by the log query request one by one. If the queried log meets the target log content, it is added to the query result set. For example, if the target log content requested by the log query request is "score:80", it is necessary to parse the logs one by one, and then traverse the logs and find out whether there is log data with a key of "score" and a value of "80" in the log. If it exists, the log is added to the query result set. In addition, when performing matching filtering, various matching filtering schemes can be adopted. For example, when the storage system supports the ability to push down operators, the filtering operation can be pushed down to the storage side for execution, thereby reducing a large amount of transmission and other overheads.

[0124] Furthermore, during the log query process, it is possible that a limited number of logs that meet the target log content cannot be queried within the timeout period. Therefore, during the log query process, the remaining available query time can be detected. If the remaining available query time is less than or equal to the preset reserved time value (ReservedTime), the current log query result and the current log query progress (for example, offset information relative to the start position of the target log storage file, etc.) are returned. The size of ReservedTime is configurable, for example, 500ms. That is, the present disclosure introduces the concept of reserved time, which is used to complete the processing of log query results and the return of log query results, so as to avoid the failure of log query due to timeout on the client side. The user can subsequently send a log query request carrying the log query progress again to obtain more log data, thereby avoiding a large number of timeout errors and improving the log query experience.

[0125] For timeout processing, in addition to reserving time, you can also use a solution to set a longer timeout. For example, if the query client wants to obtain complete log query results through a log query, you can avoid a large number of timeout errors by setting a longer timeout.

[0126] By adopting the above technical solution, the index used in the log query process includes a time index file and a data segment index file. Since the time index file indexes the log storage file by time period from the time dimension, and the data segment index file indexes the log storage file by data segment from the data segment dimension, there is no need to index by each word in the log content and the document ID of the word as in the related art. Therefore, the storage cost of the log index is greatly reduced. In addition, in the log query process, the time index file and the data segment index file can be combined to perform log queries from the time dimension and the data segment dimension respectively. Among them, the target query range within the target time range requested by the log query request can be determined through the time index file, and the target query range can be further narrowed through the data segment index file, thereby speeding up the log query process, improving the performance of the log query, and satisfying more log query usage scenarios.

[0127] In some embodiments, the log query method according to the embodiment of the present disclosure may also include: before the log is written into the log storage file, classifying the log and writing the log into the corresponding log storage file according to the classification result, different types of logs correspond to log storage files in different log sets (LogSet); and / or, before the log is written into the log storage file, parsing the log to obtain multiple key-value pairs, and then writing the log containing multiple key-value pairs into the log storage file.

[0128] The log generation pattern can be planned in advance so that various applications can generate structured log data, which is convenient for parsing log data. For example, the generated logs are in JSON, delimiter and other modes. Before writing the log data into the log storage file, the original structured log data can be parsed. After parsing, each log contains multiple key-value pairs. The logs containing multiple key-value pairs are then written into the log storage file, which makes it easier for the log search engine to process data more efficiently. The log parsing process can be completed by the application or by the log collector (LogAgent). Examples of parsing structured logs include: Fig. 9 As shown in the figure, the structured original log "24 / 06 / 26 21:06:52 Driver INFO CREATE SUCCESS" is parsed into 5 key-value pairs.

[0129] A log set is the smallest unit for log writing, consumption, and querying. Writing logs of different log types into different log sets can isolate different types of log data and avoid the problem of read amplification during log query. For example, if there are two types of logs, type 1 and type 2, and they are both written into the same log set, then during the query and scan phase of type 1 log data, it is possible that type 2 log data will be queried and scanned, resulting in additional read overhead and low log query efficiency. These problems are solved by writing logs of different log types into different log sets.

[0130] Through log classification, log structured processing and log parsing, each log finally written into the log set contains multiple parsed key-value pairs, making it easier for log search engines to process logs more efficiently.

[0131] In addition, when different log types are written into different log sets, the log set identifier of the log set can be added to the identification items for identifying the log storage file, time index file, and data segment index file in the above text. Through the log set identifier, it can be determined which log set the log storage file, time index file, data segment index file, etc. correspond to.

[0132] Fig.10 Schematic diagram of the overall architecture of log query according to an embodiment of the present disclosure. Fig.10 As shown in the figure, the overall architecture consists of three parts: the first part is log generation and collection, including log classification, log structuring and log parsing; the second part is log index generation, including the generation of time index files and data segment index files; the third part is log query. The specific implementation methods of these three parts have been described in detail in the previous article and will not be repeated here.

[0133] The following example illustrates the query performance evaluation of the log query method according to the present disclosure.

[0134] The present disclosure speeds up the log query process mainly through data block concurrency and metadata filtering based on low-cost log indexing. Assuming that there are 100 million logs, each log size is 1KB, the total original size of the 100 million logs is about 100GB. The present disclosure divides the 100GB data into 1024 100MB data blocks, and performs log queries according to a maximum concurrency of 100 between the data blocks. During the log query process, the read flow size is 4MB each time, and the average time consumed for each read is estimated to be 50 milliseconds, and the parsing and filtering time is estimated to be 30 milliseconds. In the absence of any metadata filtering, through data block concurrency, it takes about 20 seconds to complete the scanning of 100 million log data; through metadata filtering, some data segments that cannot exist can be filtered out, further speeding up the log query process, for example, 3 / 4 of the data in each data block is filtered out through metadata, and no loading and parsing and filtering are required. It takes about 5 seconds at most to complete the log query of 100 million data, which can meet most log query needs.

[0135] Fig.11 is a schematic block diagram of a log query device according to an embodiment of the present disclosure. The log query device can be applied to various log query scenarios. Fig.11 As shown, the log query device may include: a receiving module 110, which is used to receive a log query request, wherein the log query request is used to request a query for a target log content within a target time range; a log query module 120, which is used to, in response to the log query request, search for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, respectively, and search for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; a query plan is constructed according to the target query range, wherein the query plan is used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; according to the query plan, the working nodes are controlled to search in parallel from a data segment index file corresponding to the target log storage file for a data segment index that matches a data block specified by the assigned query subtask, and query the target log content based on the target data block corresponding to the data segment index, wherein the data segment index file is dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0136] By adopting the above technical solution, the index used in the log query process includes a time index file and a data segment index file. Since the time index file indexes the log storage file by time period from the time dimension, and the data segment index file indexes the log storage file by data segment from the data segment dimension, there is no need to index by each word in the log content and the document ID of the word as in the related art. Therefore, the storage cost of the log index is greatly reduced. In addition, in the log query process, the time index file and the data segment index file can be combined to perform log queries from the time dimension and the data segment dimension respectively. Among them, the target query range within the target time range requested by the log query request can be determined through the time index file, and the target query range can be further narrowed through the data segment index file, thereby speeding up the log query process, improving the performance of the log query, and satisfying more log query usage scenarios.

[0137] Optionally, the log query device also includes a time index generation module (not shown), which is used to: obtain the log of the time granularity from the log storage file according to the preset time granularity of the time index and parse to obtain the time and storage address of the log of the time granularity; generate the time index according to the storage address information of the log of the time granularity and the time identifier of the log of the time granularity determined based on the time granularity, and write the time index into the time index file in sequence, the time index corresponds one-to-one to the log of the time granularity; the time index file corresponds one-to-one to the log storage file.

[0138] Optionally, the time index generation module is further used to: if the time identifiers of the two generated time indexes are the same and the interval between the log storage addresses corresponding to the two time indexes is less than a preset merge length threshold, merge the two time indexes into one time index and update it into the time index file.

[0139] Optionally, the log query device also includes a data segment index generation module (not shown), which is used to: obtain logs from the log storage file and divide them into multiple data segments according to a preset data segment segmentation granularity, wherein each data segment contains multiple logs; for each data segment, parse the data segment to obtain a key-value pair and a log storage address for each log; for each data segment, generate a data segment index for the key of each log in the data segment, and write it into the data segment index file in the order of generation, wherein the data segment index at least indicates the position of the corresponding data segment in the log storage file.

[0140] Optionally, the data segment index generation module is further used to: when the data amount of adjacent data segments is less than a preset data segment merging threshold, aggregate the data segment indexes corresponding to the adjacent data segments into one data segment index and update the data index file.

[0141] Optionally, data segment indexes with different keys are stored and managed using different data segment index files.

[0142] Optionally, the log query module constructs a query plan based on the target query range, including: determining the number of data blocks to be accessed based on the target query range; generating a plurality of query subtasks based on the number of data blocks and the number of working nodes in an idle state, each query subtask being used to request access to a specified data block; and generating the query plan based on the plurality of query subtasks.

[0143] Optionally, the log query module queries the target log content based on the target data block corresponding to the data segment index, including: before each log data loading, first determine the data volume of the log data loaded this time, the data volume of the log data loaded this time is the minimum value between the expected value of the storage read traffic and the data volume of the target data block that has not been loaded; log data is loaded according to the data volume of the log data loaded this time.

[0144] Optionally, the log query module is further used to: detect the remaining available query time during the query process; if the remaining available query time is less than or equal to a preset reserved time value, return the current log query result and the current log query progress.

[0145] Optionally, the log query device also includes a log storage file generation module (not shown), which is used to: classify the logs before writing them to the log storage file and write them into corresponding log storage files according to the classification results, different types of logs corresponding to log storage files in different log sets; and / or, before writing them to the log storage file, parse the logs to obtain multiple key-value pairs, and then write the logs containing multiple key-value pairs into the log storage file.

[0146] Optionally, the file name of the time index file includes a log set identifier, a log time, and a unique identifier of a log storage file belonging to the log time in the corresponding log set.

[0147] The specific implementation of the operations performed by each module in the log query device according to the embodiment of the present disclosure has been described in detail in the relevant log query method, and will not be repeated here.

[0148] The present disclosure also provides a computer-readable medium having a computer program stored thereon, wherein the computer program implements the steps of any method described in the present disclosure when executed by a processing device.

[0149] The present disclosure also provides an electronic device, comprising:

[0150] a storage device having a computer program stored thereon;

[0151] A processing device is used to execute the computer program in the storage device to implement the steps of any method described in the present disclosure.

[0152] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods of the present disclosure when executed by a processor.

[0153] Reference below Fig.12 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0154] like Fig.12 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0155] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0156] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0157] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: receives a log query request, the log query request is used to request a query of a target log content within a target time range; in response to the log query request, respectively searches for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, and searches for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; constructs a query plan according to the target query range, the query plan is used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; controls working nodes according to the query plan to search in parallel for data segment indexes matching data blocks specified by the assigned query subtasks from data segment index files corresponding to the target log storage file, and queries the target log content based on the target data blocks corresponding to the data segment indexes, the data segment index files being dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0161] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, a receiving module may also be described as a "module for receiving log query requests."

[0164] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] According to one or more embodiments of the present disclosure, Example 1 provides a log query method, including: receiving a log query request, the log query request is used to request a query on a target log content within a target time range; in response to the log query request, respectively searching for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, and searching for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; constructing a query plan according to the target query range, the query plan is used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; according to the query plan, controlling working nodes to search in parallel for data segment indexes matching the data blocks specified by the assigned query subtasks from the data segment index files corresponding to the target log storage file, and querying the target log content based on the target data blocks corresponding to the data segment indexes, the data segment index files being dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0167] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the method further includes: according to the preset time granularity of the time index, obtaining the log of the time granularity from the log storage file and parsing to obtain the time and storage address of the log of the time granularity; generating the time index according to the storage address information of the log of the time granularity and the time identifier of the log of the time granularity determined based on the time granularity, and writing the time index into the time index file in sequence, the time index corresponding one-to-one to the log of the time granularity; the time index file corresponding one-to-one to the log storage file.

[0168] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the method further includes: if the time identifiers of the two generated time indexes are the same and the log storage address interval corresponding to the two time indexes is less than a preset merge length threshold, then the two time indexes are merged into one time index and updated into the time index file.

[0169] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein the method further includes: obtaining logs from the log storage file and dividing them into multiple data segments according to a preset data segment segmentation granularity, wherein each data segment contains multiple logs; for each data segment, parsing the data segment to obtain a key-value pair and a log storage address for each log; for each data segment, generating a data segment index for the key of each log in the data segment, and writing the index into the data segment index file in the order of generation, wherein the data segment index at least indicates the position of the corresponding data segment in the log storage file.

[0170] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein the method further includes: when the data amount of adjacent data segments is less than a preset data segment merging threshold, aggregating the data segment indexes corresponding to the adjacent data segments into one data segment index and updating the data index file.

[0171] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 4, wherein data segment indexes of different keys are stored and managed using different data segment index files.

[0172] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein constructing a query plan based on the target query range includes: determining the number of data blocks to be accessed based on the target query range; generating a plurality of query subtasks based on the number of data blocks and the number of working nodes in an idle state, each query subtask being used to request access to a specified data block; and generating the query plan based on the plurality of query subtasks.

[0173] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 1, wherein the querying of the target log content based on the target data block corresponding to the data segment index includes: before each log data loading, first determining the data volume of the log data loaded this time, the data volume of the log data loaded this time being the minimum value between the expected value of the storage read traffic and the data volume of the target data block that has not been loaded; and loading the log data according to the data volume of the log data loaded this time.

[0174] According to one or more embodiments of the present disclosure, Example 9 provides a method of any one of Examples 1 to 8, wherein the method further includes: detecting the remaining available query time during the query process; if the remaining available query time is less than or equal to a preset reserved time value, returning the current log query result and the current log query progress.

[0175] According to one or more embodiments of the present disclosure, Example 10 provides a method of any one of Examples 1 to 8, wherein the method further includes: before the log is written into the log storage file, classifying the log and writing the log into the corresponding log storage file according to the classification result, different types of logs correspond to log storage files in different log sets; and / or, before the log is written into the log storage file, parsing the log to obtain multiple key-value pairs, and then writing the log containing the multiple key-value pairs into the log storage file.

[0176] According to one or more embodiments of the present disclosure, Example 11 provides the method of any one of Examples 1 to 8, wherein the file name of the time index file includes a log set identifier, a log time, and a unique identifier of a log storage file belonging to the log time in the corresponding log set.

[0177] According to one or more embodiments of the present disclosure, Example 12 provides a log query device, including: a receiving module, used to receive a log query request, wherein the log query request is used to request a query for a target log content within a target time range; a log query module, used to respond to the log query request, respectively search for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, and search for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; a query plan is constructed according to the target query range, wherein the query plan is used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; according to the query plan, a working node is controlled to search in parallel from a data segment index file corresponding to the target log storage file for a data segment index matching the data block specified by the assigned query subtask, and query the target log content based on the target data block corresponding to the data segment index, wherein the data segment index file is dynamically constructed based on the data segments into which the log is segmented in the log storage file.

[0178] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-11 when executed by a processing device.

[0179] According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, including:

[0180] a storage device having a computer program stored thereon;

[0181] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-11.

[0182] According to one or more embodiments of the present disclosure, Example 15 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1 to 11 when executed by a processor.

[0183] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0184] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0185] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A log query method, characterized in that: include: Receive a log query request, where the log query request is used to request to query target log content within a target time range; In response to the log query request, the target time index and the target log storage file in the target time index file corresponding to the target time range are respectively searched from the time index file and the log storage file constructed based on the log, and the target query range corresponding to the target time index is searched from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; Constructing a query plan according to the target query range, wherein the query plan is used to indicate query subtasks corresponding to the data blocks into which the target query range is segmented; According to the query plan, the working nodes are controlled to search in parallel from the data segment index file corresponding to the target log storage file for a data segment index that matches the data block specified by the assigned query subtask, and the target log content is queried based on the target data block corresponding to the data segment index. The data segment index file is dynamically constructed based on the data segments into which the log is segmented in the log storage file.

2. The method according to claim 1, characterized in that The method further comprises: According to the preset time granularity of the time index, the log of the time granularity is obtained from the log storage file and the time and storage address of the log of the time granularity are obtained by parsing; The time index is generated according to the storage address information of the log of the time granularity and the time identifier of the log of the time granularity determined based on the time granularity, and the time index is written into the time index file in sequence, and the time index corresponds one-to-one to the log of the time granularity; the time index file corresponds one-to-one to the log storage file.

3. The method according to claim 2, characterized in that The method further comprises: If the time identifiers of the two generated time indexes are the same and the interval between the log storage addresses corresponding to the two time indexes is less than a preset merge length threshold, the two time indexes are merged into one time index and updated into the time index file.

4. The method according to claim 1, characterized in that: The method further comprises: According to a preset data segment segmentation granularity, logs are obtained from the log storage file and divided into multiple data segments, wherein each data segment contains multiple logs; For each data segment, parse the data segment to obtain the key-value pair and log storage address of each log; For each data segment, a data segment index is generated for the key of each log in the data segment, and is written into the data segment index file in the order of generation. The data segment index at least indicates the position of the corresponding data segment in the log storage file.

5. The method according to claim 4, characterized in that The method further comprises: When the data amount of adjacent data segments is less than a preset data segment merging threshold, the data segment indexes corresponding to the adjacent data segments are aggregated into one data segment index and the data index file is updated.

6. The method according to claim 4, characterized in that Data segment indexes with different keys are stored and managed using different data segment index files.

7. The method according to claim 1, characterized in that The constructing a query plan according to the target query range includes: Determining the number of data blocks to be accessed according to the target query range; Generate a plurality of query subtasks according to the number of data blocks and the number of working nodes in an idle state, each query subtask being used to request access to a specified data block; The query plan is generated according to the plurality of query subtasks.

8. The method according to claim 1, characterized in that The querying the target log content based on the target data block corresponding to the data segment index includes: Before each log data loading, the data volume of the log data to be loaded is determined, wherein the data volume of the log data to be loaded is the minimum value between the expected value of the storage read traffic and the data volume of the target data block that has not been loaded; Log data is loaded according to the data volume of the log data loaded this time.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Detecting the remaining available query time during the query process; If the remaining available query time is less than or equal to the preset reserved time value, the current log query result and the current log query progress are returned.

10. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Before the logs are written into the log storage file, the logs are classified and written into the corresponding log storage file according to the classification results, and different types of logs correspond to log storage files in different log sets; and / or, Before the log is written into the log storage file, the log is parsed to obtain a plurality of key-value pairs, and then the log including the plurality of key-value pairs is written into the log storage file.

11. The method according to any one of claims 1 to 8, characterized in that The file name of the time index file includes a log set identifier, a log time, and a unique identifier of a log storage file belonging to the log time in the corresponding log set.

12. A log query device, characterized in that: include: A receiving module, used for receiving a log query request, wherein the log query request is used for requesting to query target log content within a target time range; A log query module, configured to respond to the log query request, respectively search for a target time index and a target log storage file in a target time index file corresponding to the target time range from a time index file and a log storage file constructed based on the log, and search for a target query range corresponding to the target time index from the target log storage file; wherein the time index file is dynamically constructed based on the time in the log and corresponds one-to-one to the log storage file; construct a query plan according to the target query range, the query plan being used to indicate query subtasks corresponding to data blocks into which the target query range is segmented; According to the query plan, the working nodes are controlled to search in parallel from the data segment index file corresponding to the target log storage file for a data segment index that matches the data block specified by the assigned query subtask, and the target log content is queried based on the target data block corresponding to the data segment index. The data segment index file is dynamically constructed based on the data segments into which the log is segmented in the log storage file.

13. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Citation Information

Cited By

  • Data query method and device of WAL log, time sequence database and medium

    CN120492490A

  • Log information printing method and device and server

    CN120950016A