Log storage method, log retrieval method, device, equipment and storage medium

Abstract

The present application discloses a log storage method, a log retrieval method, an apparatus, a device, and a storage medium. The method includes obtaining a first correspondence between lexical information and a log identifier, where the lexical information is obtained by segmenting a log record, and the log identifier is identification information of the log record; according to the first correspondence, persisting the log record to a target storage space, where the target storage space includes a target index file and a log file, the target index file stores the first correspondence, and the log file stores the log record. By segmenting the log record when establishing the first correspondence, the present application can persist the log content and lexical information to the target storage space, thereby serving as the basis for subsequent full-text retrieval and enabling the log tool to support full-text retrieval.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a log storage method, a log retrieval method, a device, a device, and a storage medium. Background Art

[0002] In the cloud native technology framework, services are deployed through common container methods. For the logs generated by containers, currently, mainly Elasticsearch tool or Loki tool is used for storage and query. However, the document-based indexing method of the Elasticsearch tool is not suitable for the time-series log scenario, and since it is written in the Java language, a large number of indexes need to be resident in memory, resulting in a very high resource occupancy rate. And the label-based indexing method of the Loki tool does not support full-text retrieval. Summary of the Invention

[0003] This application provides a log storage method, a log retrieval method, a device, a device, and a storage medium to solve the technical problem that the current log tools do not support full-text retrieval.

[0004] To solve the above technical problem, in a first aspect, this application provides a log storage method, including:

[0005] Obtain a first correspondence between lexical information and a log identifier, where the lexical information is obtained by segmenting a log record, and the log identifier is the identification information of the log record;

[0006] According to the first correspondence, persist the log record to a target storage space, where the target storage space includes a target index file and a log file, the target index file stores the first correspondence, and the log file stores the log record.

[0007] When this application obtains the first correspondence between the log identifier and the lexical information obtained by segmenting the log record, it can persist the log content and the lexical information to the target storage space, thereby serving as the basis for subsequent full-text retrieval, enabling the log tool to support full-text retrieval.

[0008] In some implementation manners of the first aspect, obtaining the first correspondence between lexical information and a log identifier includes:

[0009] Establish a second correspondence between the log identifier and a sequence identifier, where the sequence identifier is the identification information of a log stream, and the log stream includes multiple log records;

[0010] Segment the log record to obtain multiple pieces of lexical information;

[0011] Generate an inverted dictionary table according to the lexical information and the second correspondence, and the inverted dictionary table stores the first correspondence.

[0012] In this implementation, by tokenizing the log records and combining with the second correspondence, an inverted dictionary table is established to obtain the correspondence among the log stream, log records, and vocabulary information, thereby serving as the index relationship chain during full-text retrieval to achieve full-text retrieval.

[0013] In some implementation manners of the first aspect, establishing the second correspondence between the log identifier and the sequence identifier includes:

[0014] Performing a hash operation on the first tag information of the log stream to obtain the sequence identifier of the log stream;

[0015] Adding the sequence identifier and the log identifier to a preset sequence table to generate an in-memory sequence table, and the in-memory sequence table stores the second correspondence.

[0016] In this implementation, through the hash operation, the uniqueness of the sequence identifier of the log stream is guaranteed, avoiding subsequent retrieval errors caused by duplicate sequence identifiers and improving the retrieval accuracy.

[0017] In some implementation manners of the first aspect, the method further includes:

[0018] Obtaining the third correspondence between the second tag information and the log identifier, where the second tag information is the tag information of the log record, and the target index file also stores the third correspondence.

[0019] In this implementation, by obtaining the third correspondence as the basis for subsequent tag retrieval, the log tool can support both full-text retrieval and tag retrieval.

[0020] In some implementation manners of the first aspect, according to the first correspondence, persisting the log records to the target storage space includes:

[0021] According to the first correspondence, storing the content information in the log records into a log file and generating a content table of the log file, and the content table stores the fourth correspondence between the log identifier and the first file offset of the log file;

[0022] Generating a target index file according to the vocabulary information and the log identifier.

[0023] In this implementation, establishing the first correspondence with the log identifier and the first file offset of the log file, so that the log file corresponding to the log identifier can be located based on the file offset during subsequent log retrieval, improving the positioning accuracy during log retrieval.

[0024] In some implementation manners of the first aspect, the target index file includes a time series sub-file and an index sub-file. Generating the target index file according to the vocabulary information and the log identifier includes:

[0025] Generate a time-series sub-file according to the timestamps and log identifiers in the log records;

[0026] Establish a fifth correspondence between the lexical information and the second file offset of the time-series sub-file to obtain an index sub-file.

[0027] In this implementation, by establishing a time-series sub-file with a time-series relationship, log storage is implemented based on the time-series relationship, so that log retrieval can be performed based on time.

[0028] In some implementation manners of the first aspect, generating a time-series sub-file according to the timestamps and log identifiers in the log records includes:

[0029] Establish a sixth correspondence between the timestamps and log identifiers in the log records to obtain a block sub-file;

[0030] Establish a seventh correspondence between the time window in the block sub-file and the third file offset of the block sub-file to obtain a time-series sub-file.

[0031] In this implementation, since the target storage space usually has multiple chunks and the log storage is performed in units of chunks, the relationship between time series and blocks is established, so that the position of the block sub-file where the log record is located can be accurately located.

[0032] In some implementation manners of the first aspect, the index sub-file also stores a sixth correspondence between the second tag information of the log record and the second file offset of the time-series sub-file.

[0033] In this implementation, the relationship between time series and tags is established as the retrieval basis for tag retrieval based on time series.

[0034] In some implementation manners of the first aspect, the method further includes:

[0035] Obtain a log stream and save the log stream to the write-ahead log system.

[0036] In this implementation, saving the log stream through the write-ahead log system avoids data loss caused by power failure or program crash and ensures data reliability.

[0037] In a second aspect, the present application also provides a log retrieval method, including:

[0038] Obtain information to be retrieved, where the information to be retrieved includes second tag information or full-text information, and the full-text information includes lexical information;

[0039] Based on the information to be retrieved, perform a log retrieval on the target storage space to obtain the target log record corresponding to the information to be retrieved. The target storage space includes a target index file and a log file. The target index file stores the correspondence between the information to be retrieved and the log identifier, and the log file stores the content information corresponding to the log identifier.

[0040] This application uses the second tag information or full-text information of the log record to support both tag retrieval and full-text retrieval simultaneously.

[0041] In some implementation manners of the second aspect, performing a log retrieval on the target storage space based on the information to be retrieved to obtain the target log record corresponding to the information to be retrieved includes:

[0042] Query the log identifier corresponding to the information to be retrieved according to the target index file;

[0043] In the content table of the log file, query the first file offset of the log file corresponding to the log identifier;

[0044] Locate the target log record corresponding to the log identifier in the log file according to the first file offset.

[0045] In some implementation manners of the second aspect, the target index file includes a time-series sub-file and an index sub-file. Querying the log identifier corresponding to the information to be retrieved according to the target index file includes:

[0046] Query the second file offset of the time-series sub-file corresponding to the information to be retrieved according to the index sub-file;

[0047] Query the log identifier corresponding to the second file offset according to the time-series sub-file.

[0048] In some implementation manners of the second aspect, the querying the second file offset of the time-series sub-file corresponding to the information to be retrieved according to the index sub-file includes:

[0049] In the index sub-file, query the fourth file offset of the record sub-file corresponding to the information to be retrieved;

[0050] In the record sub-file, query the second file offset of the time-series sub-file corresponding to the fourth file offset.

[0051] In some implementation manners of the second aspect, the querying the log identifier corresponding to the second file offset according to the time-series sub-file includes:

[0052] In the time-series sub-file, query the time window of the block sub-file corresponding to the second file offset;

[0053] In the block sub-file, query the time stamp and log identifier corresponding to the time window.

[0054] In a third aspect, the present application further provides a log storage device, including:

[0055] A first acquisition module, configured to establish a first correspondence between vocabulary information and a log identifier, where the vocabulary information is obtained by segmenting a log record, and the log identifier is the identification information of the log record;

[0056] A persistence module, configured to persist the log record to a target storage space according to the first correspondence, where the target storage space includes a target index file and a log file, the target index file stores the first correspondence, and the log file stores the log record.

[0057] In a fourth aspect, the present application further provides a log retrieval device, including:

[0058] A second acquisition module, configured to acquire information to be retrieved, where the information to be retrieved includes second tag information or full-text information, and the full-text information includes vocabulary information;

[0059] A retrieval module, configured to perform log retrieval on the target storage space based on the information to be retrieved to obtain a target log record corresponding to the information to be retrieved, where the target storage space includes a target index file and a log file, the target index file stores the correspondence between the information to be retrieved and the log identifier, and the log file stores the content information corresponding to the log identifier.

[0060] In a fifth aspect, the present application further provides a computer device, including a processor and a memory, where the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the log storage method in the first aspect or the log retrieval method in the second aspect.

[0061] In a sixth aspect, the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by the processor, it implements the log storage method in the first aspect or the log retrieval method in the second aspect.

[0062] In a seventh aspect, the present application further provides a computer program product, and when the computer program product runs on a computer device, it causes the computer device to implement the log storage method in the first aspect or the log retrieval method in the second aspect when executed.

[0063] It should be noted that for the beneficial effects of the above third aspect to the seventh aspect, please refer to the relevant descriptions of the above first aspect or the second aspect, and will not be elaborated here. Description of the Drawings

[0064] Figure 1 Schematic flowchart of the log storage method shown in the embodiments of the present application;

[0065] Figure 2 Schematic flowchart of the specific process of step S101 shown in the embodiments of the present application;

[0066] Figure 3 Schematic flowchart of the specific process of step S102 shown in the embodiments of the present application;

[0067] Figure 4 Schematic diagram of the read operation and write operation shown in the embodiments of the present application;

[0068] Figure 5 Schematic diagram of the log file shown in the embodiments of the present application;

[0069] Figure 6 Schematic diagram of the block sub-file shown in the embodiments of the present application;

[0070] Figure 7 Schematic diagram of the timing sub-file shown in the embodiments of the present application;

[0071] Figure 8 Schematic diagram of the record sub-file shown in the embodiments of the present application;

[0072] Figure 9 Schematic diagram of the index sub-file shown in the embodiments of the present application;

[0073] Figure 10 Schematic flowchart of the log retrieval method shown in the embodiments of the present application;

[0074] Figure 11 Schematic diagram of the structure of the log storage device shown in the embodiments of the present application;

[0075] Figure 12 Schematic diagram of the structure of the log retrieval device shown in the embodiments of the present application;

[0076] Figure 13 Schematic diagram of the structure of the computer device shown in the embodiments of the present application. Detailed implementation manners

[0077] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0078] As described in the related art, the document - based indexing method of the Elasticsearch tool is not suitable for the time - series log scenario. Moreover, since it is written in the Java language, a large number of indexes need to be resident in memory, resulting in a very high resource occupancy rate. The Loki tool's label - based indexing method cannot support full - text search.

[0079] Therefore, in this application, when establishing the first correspondence relationship, the log records are tokenized so that the log content and vocabulary information can be persisted to the target storage space, thereby serving as the basis for subsequent full - text search and enabling the log tool to support full - text search.

[0080] Please refer to Figure 1 , Figure 1 FIG. [FIG. number not provided in the original, so it's left as is] is a schematic flowchart of a log storage method provided by an embodiment of this application. The log storage method of the embodiment of this application can be applied to a computer device, which includes but is not limited to devices such as smart phones, laptops, tablets, desktop computers, physical servers, and cloud servers. As Figure 1 shown, the log storage method of this embodiment includes steps S101 to S102, which are described in detail as follows:

[0081] Step S101, obtain a first correspondence relationship between vocabulary information and a log identifier, where the vocabulary information is obtained by tokenizing log records, and the log identifier is the identification information of the log records.

[0082] In this step, the log records are log information in a log stream. The log stream (LogStream) is the basic unit of log reading and writing, which can facilitate the classified storage of logs. At the same time, the reading and writing method of the log stream can minimize the number of reads and writes and improve the log processing efficiency. Each log record in the log stream includes but is not limited to a timestamp and content information.

[0083] Exemplarily, the first label information of the log stream is {app = nginx, instance = "1.1.1.1"}, and a log record in this log stream is {"ts": "2021 - 12 - 30T17:50:33.2633687+08:00", "line": "XXis an enterprise - level PaaS platform for large and super large enterprises", where ts is the timestamp and line is the content information.

[0084] Optionally, for the log storage engine of cloud-native technology, the log stream can be the log stream of Kubernetes logs, and the Kubernetes logs include but are not limited to Kubernetes pods and Kubernetes events.

[0085] As an alternative implementation, as Figure 4 shown, this embodiment processes the log stream in the form of a first in-memory table (including establishing a first correspondence relationship), where the first in-memory table is the memory structure (memtable) in the computer device for processing the log stream, and it allows writing and reading. Optionally, when the data written to the first in-memory table reaches a preset upper limit, to avoid data processing blockage, it is automatically converted into a second in-memory table (immumemtable), that is, if the first in-memory table reaches the upper limit value of the memory space, the log stream is stored in the second in-memory table.

[0086] It should be noted that the process of establishing the first correspondence relationship can be executed on this computer device; it can also be executed on other computer devices and then the computer file corresponding to the first correspondence relationship is transplanted to this computer device. It can be understood that this computer file can be represented as the first in-memory table, or it can be other table forms after data conversion of the first in-memory table. For example, the first in-memory table is converted into a database table and the database table is transplanted to this computer device.

[0087] The inverted index table is an inverted index table that determines the record position based on the attribute value, and each item of it includes an attribute value and the record addresses of this attribute value. The dictionary inverted index table is an inverted index table that determines the log identifier based on the lexical information as the attribute value. The log identifier is the identification information representing the uniqueness of the log record, such as the log ID (Identity document); the lexical information is obtained by tokenizing the log content. Optionally, the tokenization method can be to tokenize the log content through any tokenization tool (such as a tokenization model constructed based on a convolutional neural network).

[0088] Optionally, the log storage method of this embodiment is written based on the GoLang language. Compared with the Java language, the GoLang language is lighter and can effectively reduce the resource occupancy rate.

[0089] In a possible implementation manner, the log stream is obtained and directly written into the memtable. In another possible implementation manner, as Figure 4As shown, since the memtable is a temporary storage and data will be lost when the computer device loses power or the program crashes, the log stream is obtained and saved to the write-ahead log system (Wal) based on the computer write operation. When the program restarts, the log stream of the Wal is read and rewritten to the memtable to ensure data reliability.

[0090] Step S102: Persist the log record to the target storage space according to the first correspondence relationship. The target storage space includes a target index file and a log file. The target index file stores the first correspondence relationship, and the log file stores the log record.

[0091] In this step, the target storage space is a readable storage medium with data persistence capabilities, such as a disk (including a computer hard disk and a computer floppy disk), etc. Persistence is a processing mechanism that converts transient data into persistent data to permanently retain the data.

[0092] The target index file stores the first correspondence relationship between the vocabulary information and the log identifier, so that during subsequent log retrieval, based on the full-text information composed of the vocabulary information, full-text retrieval can be performed according to this first correspondence relationship to determine the log identifier corresponding to the vocabulary information to be retrieved, and then the corresponding log content can be read from the log file according to this log identifier. When the log file stores the log content in the log record, the correspondence relationship between the log identifier of the log record and the log content is also recorded to facilitate accurate reading of the log content later.

[0093] It should be noted that in this embodiment, by segmenting the log record when establishing the first correspondence relationship, when the log record is persisted to the target storage space, the target storage space can also record the vocabulary information as the basis for subsequent full-text retrieval, enabling the log tool to support full-text retrieval.

[0094] In some embodiments, on the basis of Figure 1 the embodiment shown, Figure 2 a specific flowchart of step S101 is shown. As Figure 2 shown, the above step S101 includes:

[0095] Step S201: Establish a second correspondence relationship between the log identifier and the sequence identifier. The sequence identifier is the identification information of the log stream, and the log stream includes multiple log records.

[0096] Step S202: Segment the log record to obtain multiple vocabulary information.

[0097] Step S203: Generate an inverted dictionary table according to the lexical information and the second correspondence relationship. The inverted dictionary table stores the first correspondence relationship.

[0098] In this embodiment, the first memory table in the optional implementation manner of the above step S101 includes a memory sequence table (memseries) and an inverted dictionary table (termposting). The memory sequence table is the basic table in the first memory table, which records the second correspondence relationship between the log identifier and the sequence identifier, that is, records the correspondence relationship between each log record and the log stream.

[0099] Optionally, the generation process of the memory sequence table includes: performing a hash operation on the first tag information of the log stream to obtain the sequence identifier of the log stream; adding the sequence identifier and the log identifier to a preset sequence table to generate a memory sequence table, and the memory sequence table stores the second correspondence relationship.

[0100] In this optional embodiment, in a possible implementation manner, perform a hash operation on the first tag information of the log stream to generate a sequence identifier (seriesid) that uniquely identifies the log stream. Query whether the seriesid of the log stream exists in the preset sequence table. If the seriesid of the log stream does not exist in the preset sequence table, add the seriesid to the preset sequence table, and at the same time apply for a memory block for storing the log identifier (logid), and add the logid to the preset sequence table to obtain the memory sequence table; if the seriesid of the log stream exists in the preset sequence table, apply for a memory block for storing the log identifier (logid), and add the logid to the preset sequence table to obtain the memory sequence table.

[0101] Exemplarily, the memory sequence table is shown in Table 1 below:

[0102]

[0103] In another possible implementation manner, the time information of the log is also considered, that is, while applying for a memory block for storing the log identifier (logid), number each log record as (timestamp, logid), and add it to the preset sequence table to obtain the memory sequence table.

[0104] Exemplarily, the memory sequence table is shown in Table 2 below:

[0105]

[0106] Exemplarily, based on Table 2 above, the inverted dictionary table is shown in Table 3 below:

[0107]

[0108] Optionally, the method further includes: obtaining a third correspondence between the second tag information and the sequence identifier, where the second tag information is the tag information of the log record, and the third correspondence is further stored in the target index file.

[0109] In this embodiment, the first memory table further includes a series posting, which is an inverted index table that determines a log identifier using the second tag information as an attribute value. The second tag information is a keyword of the log record, which can be a time keyword, a log name keyword, etc. Optionally, the second tag information is represented in the form of a key-value pair, i.e., labelname-value. Exemplarily, based on the above

[0110] Table 2, the series posting is shown in Table 4 below:

[0111]

[0112] In some embodiments, based on Figure 1 the embodiment shown, Figure 3 a specific process schematic diagram of step S102 is shown. As Figure 3 shown, step S102 includes:

[0113] Step S301, according to the first correspondence, store the content information in the log record into the log file, and generate a content table of the log file, where the content table stores a fourth correspondence between the log identifier and the first file offset of the log file.

[0114] Step S302, generate the target index file according to the vocabulary information and the log identifier.

[0115] In this embodiment, the file offset is the number of bytes moved forward or backward from a specified position in the file, which is used to query target data from the file. Since the log file stores log content, and there are multiple log data in the log content of each log record corresponding to the second tag information, in order to accurately locate the corresponding log data of the log content, the log identifier is associated with the first file offset.

[0116] Exemplarily, the log file (logs) is as Figure 5 shown, log represents the log content line in the example of step S101 above, TOC is the content table, ref is the first file offset, and log_1_ref is the first file offset of the first logs file.

[0117] Optionally, the target index file includes, but is not limited to, an index sub-file, a posting sub-file, a series sub-file, and a chunk sub-file. The index sub-file is used to store index information of lexical information. The posting sub-file is used to store index information of the series associated with a certain label or term. The series sub-file is used for index information of the labels and chunks included in each series. The chunk sub-file is used to store the timestamp and log identifier of each log.

[0118] In some embodiments, based on the embodiments shown Figure 3 Step S302 includes: generating the series sub-file according to the timestamp and the log identifier in the log record; establishing a fifth correspondence between the lexical information and the second file offset of the series sub-file to obtain the index sub-file.

[0119] In this embodiment, for the series sub-file, in a possible implementation manner, a correspondence between the timestamp and the log identifier is established to obtain the series sub-file. Since the log file also stores the log identifier, there is an association relationship between the series sub-file and the log file, which is convenient for the subsequent relationship chain of log retrieval.

[0120] In another possible implementation manner, a sixth correspondence between the timestamp in the log record and the log identifier is established to obtain the chunk sub-file; a seventh correspondence between the time window in the chunk sub-file and the third file offset of the chunk sub-file is established to obtain the series sub-file. The chunk sub-file is the chunk sub-file. The disk or other readable storage medium is regarded as a Block, and a Block is an independent small database that stores all the information for log retrieval for a period of time, such as label and lexical information, etc. The chunk sub-file is a storage structure in the Block. The size of all chunk sub-files on the disk is not greater than 512M, so multiple chunk sub-files are required to store the index information of the timestamp and the log identifier.

[0121] Exemplarily, Figure 6 A schematic diagram of the chunk sub-file is shown. For each chunk sub-file, it includes a sixth correspondence between the timestamp ts and the log identifier logid. Figure 7A schematic diagram of the timing sub-file is shown. The timing sub-file stores the second tag information corresponding to each timing series and the index information of the chunks. Among them, label_name and label_value are the key-value pairs of the second tag information, Mint and Maxt are the time windows of the chunks, which respectively identify the minimum time and the maximum time of the log, and chunk_ref is the file offset of the chunk sub-file.

[0122] For the index sub-file, in a possible implementation manner, a fifth correspondence is established between the vocabulary information and the second file offset of the timing sub-file to obtain the index sub-file. Since there is an association relationship between the timing sub-file and the log file, associating the vocabulary information with the timing sub-file can obtain the association relationship between the vocabulary information and the log file.

[0123] In another possible implementation manner, the index sub-file also stores a sixth correspondence between the second tag information of the log record and the second file offset of the timing sub-file. Optionally, a record sub-file storing all the timing sub-files associated with the second tag information is established, and then a correspondence is established between the second tag information or the vocabulary information and the file offset of the record sub-file to obtain the index sub-file.

[0124] Exemplarily, as Figure 8 shown in the schematic diagram of the record sub-file, series_ref is the file offset of the timing sub-file, and len<4b> represents the label label. As Figure 9 shown in the schematic diagram of the index sub-file, term is the vocabulary information, and multiple vocabulary information constitutes the full-text information message.

[0125] Please refer to Figure 10 , Figure 10 which is a schematic flowchart of a log retrieval method provided by an embodiment of the present application. The log retrieval method of the embodiment of the present application can be applied to computer devices, including but not limited to devices such as smart phones, laptop computers, tablet computers, desktop computers, physical servers, and cloud servers. As Figure 10 shown, the log retrieval method of this embodiment includes steps S1001 to step S102, which are described in detail as follows:

[0126] Step S1001, obtain the information to be retrieved. The information to be retrieved includes the second tag information or the full-text information, and the full-text information includes the vocabulary information.

[0127] Step S1002: Based on the information to be retrieved, perform a log retrieval on the target storage space to obtain the target log record corresponding to the information to be retrieved. The target storage space includes a target index file and a log file. The target index file stores the correspondence between the information to be retrieved and the log identifier, and the log file stores the content information corresponding to the log identifier.

[0128] In some embodiments, based on Figure 10 the embodiments shown, step S1002 includes:

[0129] Query the log identifier corresponding to the information to be retrieved according to the target index file;

[0130] In the content table of the log file, query the first file offset of the log file corresponding to the log identifier;

[0131] Locate the target log record corresponding to the log identifier in the log file according to the first file offset.

[0132] In some embodiments, the target index file includes a time-series sub-file and an index sub-file. Querying the log identifier corresponding to the information to be retrieved according to the target index file includes:

[0133] Query the second file offset of the time-series sub-file corresponding to the information to be retrieved according to the index sub-file;

[0134] Query the log identifier corresponding to the second file offset according to the time-series sub-file.

[0135] In some embodiments, the querying the second file offset of the time-series sub-file corresponding to the information to be retrieved according to the index sub-file includes:

[0136] In the index sub-file, query the fourth file offset of the record sub-file corresponding to the information to be retrieved;

[0137] In the record sub-file, query the second file offset of the time-series sub-file corresponding to the fourth file offset.

[0138] In some embodiments, the querying the log identifier corresponding to the second file offset according to the time-series sub-file includes:

[0139] In the time-series sub-file, query the time window of the block sub-file corresponding to the second file offset;

[0140] In the block sub-file, query the time stamp and the log identifier corresponding to the time window.

[0141] It is understandable that this embodiment is a method for retrieving logs in a target storage space based on the log storage method of the corresponding embodiment above. It has the corresponding features of the log storage method. For specific details, please refer to the corresponding description above. Figures 1 to 9 To improve the retrieval efficiency, as shown in Figure 4 , based on computer read operations, the target index file and log file in the target storage space are cached in the buffer in the computer device memory, and the above retrieval method is implemented in this memory space. Figures 1 to 9 It should be noted that Figure 4 As an example rather than a limitation, for example, the target index file includes but is not limited to index sub-files, posting sub-files, series sub-files, and chunk sub-files, and the log file is a logs file. Based on the log second tag information (labelname and labelvalue) or full text information (message and term), in the index sub-file of Figure 9 , the posting_ref corresponding to the second tag information or full text information is queried; then through this posting_ref in the posting sub-file of Figure 8 , one or more series_ref are queried; then through series_ref in the series sub-file of Figure 7 , one or more chunk_ref are queried; then through chunk_ref in the chunk sub-file of Figure 6 , the timestamp and log identifier (ts, logid) are queried; finally, through logid, the log content is located in the logs file of Figure 5 .

[0142] As an example rather than a limitation, Figures 5 to 9 For example, the target index file includes but is not limited to index sub-files, posting sub-files, series sub-files, and chunk sub-files, and the log file is a logs file. Based on the log second tag information (labelname and labelvalue) or full text information (message and term), in the index sub-file of Figure 9 , the posting_ref corresponding to the second tag information or full text information is queried; then through this posting_ref in the posting sub-file of Figure 8 , one or more series_ref are queried; then through series_ref in the series sub-file of Figure 7 , one or more chunk_ref are queried; then through chunk_ref in the chunk sub-file of Figure 6 , the timestamp and log identifier (ts, logid) are queried; finally, through logid, the log content is located in the logs file of Figure 5 . Figure 9 the index sub-file of Figure 9 , the posting_ref corresponding to the second tag information or full text information is queried; then through this posting_ref in Figure 8 the posting sub-file of Figure 8 , one or more series_ref are queried; then through series_ref in Figure 7 the series sub-file of Figure 7 , one or more chunk_ref are queried; then through chunk_ref in Figure 6 the chunk sub-file of Figure 6 , the timestamp and log identifier (ts, logid) are queried; finally, through logid in Figure 5 the logs file of Figure 5 , the log content is located.

[0143] To execute the log storage method corresponding to the above method embodiment to achieve the corresponding functions and technical effects. Refer to Figure 11 Figure 11 As shown in Figure 11 , a structural block diagram of a log storage device provided by an embodiment of the present application is shown. For ease of description, only the parts related to this embodiment are shown. The log storage device provided by the embodiment of the present application includes:

[0144] A first acquisition module 1101, configured to acquire a first correspondence relationship between vocabulary information and a log identifier, where the vocabulary information is obtained by segmenting a log record, and the log identifier is the identification information of the log record;

[0145] ​A persistence module 1102 is configured to persist the log records into a target storage space according to the first correspondence relationship. The target storage space includes a target index file and a log file. The target index file stores the first correspondence relationship, and the log file stores the log records.

[0146] In some embodiments, the first memory table further includes a memory sequence table. The acquisition module 1101 includes:

[0147] A establishing sub-module is configured to establish a second correspondence relationship between the log identifier and the sequence identifier. The sequence identifier is the identification information of the log stream, and the log stream includes a plurality of log records.

[0148] A word segmentation sub-module is configured to perform word segmentation on the log records to obtain a plurality of lexical information.

[0149] A generating sub-module is configured to generate an inverted index table of the dictionary according to the lexical information and the second correspondence relationship. The inverted index table of the dictionary stores the first correspondence relationship.

[0150] In some embodiments, the establishing sub-module includes:

[0151] An operation unit is configured to perform a hash operation on the first tag information of the log stream to obtain the sequence identifier of the log stream.

[0152] An adding unit is configured to add the sequence identifier and the log identifier to a preset sequence table to generate a memory sequence table. The memory sequence table stores the second correspondence relationship.

[0153] In some embodiments, the log storage device of the establishing module further includes:

[0154] A establishing module is configured to establish a third correspondence relationship between the second tag information and the sequence identifier. The second tag information is the tag information of the log record, and the target index file further stores the third correspondence relationship.

[0155] In some embodiments, the persistence module 1102 includes:

[0156] A storage sub-module is configured to store the content information in the log records into the log file according to the first correspondence relationship, and generate a content table of the log file. The content table stores a fourth correspondence relationship between the log identifier and the first file offset of the log file.

[0157] A third establishing sub-module is configured to generate the target index file according to the lexical information and the log identifier.

[0158] In some embodiments, the target index file includes a timing sub-file and an index sub-file, and the third establishment sub-module includes:

[0159] A generation unit, configured to generate the timing sub-file according to the timestamp in the log record and the log identifier;

[0160] An establishment unit, configured to establish a fifth correspondence between the vocabulary information and the second file offset of the timing sub-file to obtain the index sub-file.

[0161] In some embodiments, the generation unit includes:

[0162] A first establishment sub-unit, configured to establish a sixth correspondence between the timestamp in the log record and the log identifier to obtain a block sub-file;

[0163] A second establishment sub-unit, configured to establish a seventh correspondence between the time window in the block sub-file and the third file offset of the block sub-file to obtain the timing sub-file.

[0164] In some embodiments, the index sub-file further stores a sixth correspondence between the second tag information of the log record and the second file offset of the timing sub-file.

[0165] The above log storage device can implement the log storage method in the above method embodiments. The optional items in the above method embodiments are also applicable to this embodiment and will not be elaborated here. The remaining content of the embodiments of the present application can refer to the content of the above method embodiments and will not be repeated in this embodiment.

[0166] To execute the corresponding log retrieval method in the above method embodiments to achieve the corresponding functions and technical effects. Refer to Figure 12 , Figure 12 FIG. shows a structural block diagram of a log retrieval device provided by an embodiment of the present application. For ease of description, only the parts related to this embodiment are shown. The log retrieval device provided by the embodiment of the present application includes:

[0167] A second acquisition module 1201, configured to acquire retrieval information, where the information to be retrieved includes second tag information or full-text information, and the full-text information includes vocabulary information;

[0168] A retrieval module 1202, configured to perform a log retrieval on a target storage space based on the information to be retrieved to obtain a target log record corresponding to the information to be retrieved. The target storage space includes a target index file and a log file. The target index file stores a correspondence between the retrieval information and the log identifier, and the log file stores content information corresponding to the log identifier.

[0169] In some embodiments, based on the Figure 12 embodiment shown, the retrieval module 1202 includes:

[0170] A first query sub-module, configured to query, in the target index file, a log identifier corresponding to the information to be retrieved;

[0171] A second query sub-module, configured to query, in the content table of the log file, a first file offset of the log file corresponding to the log identifier;

[0172] A positioning sub-module, configured to, according to the first file offset, locate a target log record corresponding to the log identifier in the log file.

[0173] In some embodiments, the target index file includes a timing sub-file and an index sub-file, and the first query sub-module includes:

[0174] A first query unit, configured to query, in the index sub-file, a second file offset of the timing sub-file corresponding to the information to be retrieved;

[0175] A second query unit, configured to query, in the timing sub-file, a log identifier corresponding to the second file offset.

[0176] In some embodiments, the second query unit includes:

[0177] A first query subunit, configured to query, in the timing sub-file, a time window of a block sub-file corresponding to the second file offset;

[0178] A second query z subunit, configured to query, in the block sub-file, a time stamp and a log identifier corresponding to the time window.

[0179] The above log retrieval device can implement the log retrieval method in the above method embodiment. The optional items in the above method embodiment are also applicable to this embodiment and will not be elaborated here.

[0180] Figure 13 It is a schematic structural diagram of a computer device provided in an embodiment of the present application. As Figure 13 shown, the computer device 13 in this embodiment includes: at least one processor 130 ( Figure 13 only one is shown in the figure), a processor, a memory 131, and a computer program 132 stored in the memory 131 and executable on the at least one processor 130. When the processor 130 executes the computer program 132, the steps in any of the above method embodiments are implemented.

[0181] The computer device 13 may be a computing device such as a smart phone, a tablet computer, a desktop computer, and a cloud server. The computer device may include, but is not limited to, a processor 130 and a memory 131. Those skilled in the art can understand that, Figure 13 merely examples of the computer device 13, which do not constitute a limitation on the computer device 13, may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0182] The so-called processor 130 may be a central processing unit (CPU), and the processor 130 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0183] The memory 131 may be an internal storage unit of the computer device 13 in some embodiments, such as the hard disk or memory of the computer device 13. The memory 131 may also be an external storage device of the computer device 13 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 13. Further, the memory 131 may also include both the internal storage unit and the external storage device of the computer device 13. The memory 131 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 131 may also be used to temporarily store data that has been output or will be output.

[0184] In addition, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0185] An embodiment of the present application provides a computer program product. When the computer program product runs on a computer device, it enables the computer device to execute the steps in the above-mentioned method embodiments.

[0186] In several embodiments provided by the present application, it can be understood that each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the block may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.

[0187] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs.

[0188] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present application. It should be understood that the above descriptions are only specific embodiments of the present application and are not used to limit the protection scope of the present application. In particular, for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A log storage method, characterized in that, Including: Obtain a first correspondence relationship between vocabulary information and a log identifier, where the vocabulary information is obtained by word segmenting a log record, and the log identifier is the identifier information of the log record; According to the first correspondence relationship, persist the log record to a target storage space, where the target storage space includes a target index file and a log file, the target index file stores the first correspondence relationship, and the log file stores the log record; The obtaining of the first correspondence relationship between vocabulary information and a log identifier includes: Establish a second correspondence relationship between the log identifier and a sequence identifier, where the sequence identifier is the identifier information of a log stream, and the log stream includes multiple log records; Perform word segmentation on the log record to obtain multiple pieces of vocabulary information; Generate an inverted dictionary table according to the vocabulary information and the second correspondence relationship, where the inverted dictionary table stores the first correspondence relationship.

2. The log storage method according to claim 1, characterized in that, The establishing of the second correspondence relationship between the log identifier and a sequence identifier includes: Perform a hash operation on the first tag information of the log stream to obtain the sequence identifier of the log stream; Add the sequence identifier and the log identifier to a preset sequence table to generate an in-memory sequence table, where the in-memory sequence table stores the second correspondence relationship.

3. The log storage method according to claim 1, characterized in that, The method further includes: Obtain a third correspondence relationship between second tag information and a sequence identifier, where the second tag information is the tag information of the log record, the sequence identifier is the identifier information of a log stream, and the target index file also stores the third correspondence relationship.

4. The log storage method according to claim 1, characterized in that, The persisting of the log record to the target storage space according to the first correspondence relationship includes: According to the first correspondence relationship, store the content information in the log record to the log file, and generate a content table of the log file, where the content table stores a fourth correspondence relationship between the log identifier and the first file offset of the log file; Generate the target index file according to the vocabulary information and the log identifier.

5. The log storage method according to claim 4, wherein The target index file includes a time series sub-file and an index sub-file, and the generating of the target index file according to the vocabulary information and the log identifier includes: Generate the time series sub-file according to the timestamp in the log record and the log identifier; Establish a fifth correspondence relationship between the vocabulary information and the second file offset of the time series sub-file to obtain the index sub-file.

6. The log storage method according to claim 5, characterized in that The generating of the time series sub-file according to the timestamp in the log record and the log identifier includes: Establish a sixth correspondence relationship between the timestamp in the log record and the log identifier to obtain a block sub-file; Establish a seventh correspondence relationship between the time window in the block sub-file and the third file offset of the block sub-file to obtain the time series sub-file.

7. The log storage method according to claim 5, wherein The index sub-file also stores a sixth correspondence relationship between the second tag information of the log record and the second file offset of the time series sub-file.

8. A log retrieval method, characterized in that, Including: Obtain information to be retrieved, where the information to be retrieved includes second tag information or full text information, and the full text information includes vocabulary information; Based on the information to be retrieved, perform log retrieval on the target storage space to obtain the target log record corresponding to the information to be retrieved. The target storage space includes a target index file and a log file. The target index file stores the correspondence between the information to be retrieved and the log identifier, and the log file stores the content information corresponding to the log identifier; The target storage space stores the log records persisted according to the first correspondence of the log retrieval method described in any one of claims 1 to 7.

9. A log storage device, characterized in that, Comprising: A first acquisition module, configured to acquire a first correspondence between vocabulary information and a log identifier, where the vocabulary information is obtained by segmenting a log record, and the log identifier is the identifier information of the log record; A persistence module, configured to persist the log record to the target storage space according to the first correspondence. The target storage space includes a target index file and a log file. The target index file stores the first correspondence, and the log file stores the log record; The acquisition module includes: A creation sub-module, configured to create a second correspondence between the log identifier and a sequence identifier, where the sequence identifier is the identifier information of a log stream, and the log stream includes multiple log records; A segmentation sub-module, configured to segment the log record to obtain multiple pieces of vocabulary information; A generation sub-module, configured to generate an inverted dictionary table according to the vocabulary information and the second correspondence. The inverted dictionary table stores the first correspondence.

10. A log retrieval device, characterized in that, Comprising: A second acquisition module, configured to acquire information to be retrieved, where the information to be retrieved includes second tag information or full-text information, and the full-text information includes vocabulary information; A retrieval module, configured to perform log retrieval on the target storage space based on the information to be retrieved to obtain the target log record corresponding to the information to be retrieved. The target storage space includes a target index file and a log file. The target index file stores the correspondence between the information to be retrieved and the log identifier, and the log file stores the content information corresponding to the log identifier; The target storage space stores the log records persisted according to the first correspondence of the log retrieval method described in any one of claims 1 to 7.

11. A computer device, characterized in that, Comprising a processor and a memory. The memory is used to store a computer program. When the computer program is executed by the processor, it implements the log storage method described in any one of claims 1 to 7, or the log retrieval method described in claim 8.

12. A computer-readable storage medium, characterized in that, It stores a computer program. When the computer program is executed by a processor, it implements the log storage method described in any one of claims 1 to 7, or the log retrieval method described in claim 8.

13. A computer program product, characterized in that, When the computer program product runs on a computer device, it causes the computer device to implement the log storage method described in any one of claims 1 to 7, or the log retrieval method described in claim 8.

Images