A log compression and index construction method and system

Abstract

The application provides a log compression and index construction method and system, wherein the method comprises the following steps: obtaining an original log, and dividing the original log into an original template and an original variable based on a log analysis tool; counting the frequency of occurrence of the original variable in the original log, and dividing the original variable into a variable with an inverted index and a variable without an inverted index according to the frequency of occurrence; constructing an inverted index and a corresponding first storage unit based on the variable with the inverted index, and constructing an abstract index and a corresponding second storage unit based on the variable without the inverted index; compressing the inverted index and the abstract index to obtain compressed indexes; and compressing the first storage unit and the second storage unit to obtain compressed log data, which can quickly exclude irrelevant data and reduce invalid decompression during large-scale log retrieval, thereby significantly reducing retrieval delay while maintaining high log compression rate, and achieving the best balance between large-scale log storage cost and query efficiency.

Description

Technical Field

[0001] This invention relates to the field of log processing technology, and in particular to a log compression and index building method and system. Background Technology

[0002] In modern cloud systems, log data has become a critical data source supporting scenarios such as anomaly detection, security checks, and system modeling. With the continuous expansion of cloud service scale, the daily log volume generated by the system has reached petabyte (PB) levels. To control storage costs, high-density compression of logs is typically required. To achieve rapid location within massive compressed logs, building compact and efficient indexes to avoid high-latency brute-force searches has become a core requirement.

[0003] Existing log indexing schemes typically build an inverted index on all extracted variable locations after parsing the log. Since the variable portion is the only encoded and compressed content in the log, this scheme theoretically guarantees index compactness. However, in practical applications, it has been found that traditional inverted indexes face severe index bloat problems when processing large-scale logs, with the index space potentially exceeding the original text space.

[0004] Analysis revealed that the root cause of index bloat lies in the prevalence of numerous low-frequency variables (such as variables that appear only once in the entire index space) in the logs. For these variables, building an inverted index requires not only maintaining a massive variable dictionary but also recording the corresponding inverted list position information. This structured recording method incurs metadata overhead far exceeding the storage overhead of the original variable text for low-frequency variables; simultaneously, the frequent decompression of the large variable dictionary during queries significantly impacts retrieval efficiency. Therefore, how to build a compact index that can accommodate both low- and high-frequency variable characteristics and support low-latency queries while maintaining a high log compression ratio has become a key challenge for large-scale log storage systems. Summary of the Invention

[0005] This invention provides a log compression and indexing method and system to solve the problems of index bloat and low storage efficiency caused by building inverted indexes on all variables in the prior art. By classifying the original variables by frequency and adopting differentiated indexing and storage strategies, this invention can effectively reduce the index size and storage overhead while maintaining a high log compression rate. At the same time, it significantly reduces the retrieval latency for large-scale compressed logs through an efficient filtering mechanism.

[0006] This invention provides a log compression and index building method, the method comprising: Obtain the raw logs and divide them into raw templates and raw variables based on the log parsing tool; The frequency of occurrence of the original variables in the original log is counted, and the original variables are divided into variables with inverted indexes and variables without inverted indexes based on the frequency of occurrence. An inverted index and its corresponding first storage unit are constructed based on the variables with inverted indexes, and a summary index and its corresponding second storage unit are constructed based on the variables without inverted indexes. The inverted index and the digest index are compressed to obtain a compressed index; the first storage unit and the second storage unit are compressed to obtain compressed log data.

[0007] According to the log compression and index construction method provided by the present invention, the method further includes a log retrieval step: receiving a log retrieval request containing a string to be retrieved; performing constant matching on the string to be retrieved based on the information of the original template and the log retrieval request to obtain a sequence of storage units to be retrieved; filtering the sequence of storage units to be retrieved based on the compressed index to obtain a sequence of storage units to be decompressed; decompressing the sequence of storage units to be decompressed, and reconstructing the target log corresponding to the string to be retrieved by combining it with the information of the original template.

[0008] According to the log compression and index construction method provided by the present invention, the step of dividing the original variables into variables with inverted indexes and variables without inverted indexes based on the occurrence frequency includes: classifying the original variables whose occurrence frequency is greater than the frequency threshold as variables with inverted indexes; and classifying the original variables whose occurrence frequency is less than or equal to the frequency threshold as variables without inverted indexes.

[0009] According to the log compression and index construction method provided by the present invention, the inverted index includes a dictionary and an inverted list. The construction method includes: constructing the dictionary based on the set of values ​​of the variable with the inverted index, and mapping the variable values ​​to integer numbers; wherein the mapping maps values ​​with lower lexicographical order to smaller integer numbers; for each value of the variable with the inverted index, recording its position information in the original log to form the inverted list.

[0010] According to the log compression and index construction method provided by the present invention, the method for constructing the summary index includes: arranging the variables without inverted indexes into a variable list according to their occurrence positions in the original log; cutting the variable list into a fixed-length segment according to a preset length to obtain a set of segmented variable lists; extracting data features from each variable list in the set of segmented variable lists to construct a data summary; combining the various data summaries to obtain a data summary set, and using the data summary set as the summary index.

[0011] According to the log compression and index construction method provided by the present invention, the step of extracting data features from each variable list in the segmented variable list set and constructing a data summary includes: Identify the type of the segmented variable list and classify it into a pure numeric variable list and a character variable list; For the list of purely numeric variables, the range of numbers is recorded as its data summary; For the list of character variables, the variable length range and the set of character values ​​are recorded as its data summary; wherein, the set of character values ​​includes a set of required characters and a set of possible characters.

[0012] According to the log compression and index construction method provided by the present invention, the construction method of the first storage unit and the second storage unit includes: For the variables with inverted indexes, they are stored in a log table in dictionary form, and the log table is used to construct the first storage unit; For the variables without inverted indexes, store them in plain text in a variable list, and use the variable list to construct the second storage unit.

[0013] According to the log compression and index construction method provided by the present invention, the method for constructing the log table includes: arranging the variable portions of the original log in sequence to form an original log list; cutting the original log list according to a preset length to form a set of cut original log lists; for each list in the set of cut original log lists, replacing the variable items belonging to the inverted index variable with the integer number of the variable in the dictionary, and replacing the variable items belonging to the non-inverted index variable with placeholders; and using the cut original log list after the replacement as the log table.

[0014] According to the log compression and index construction method provided by the present invention, the step of filtering the sequence of storage units to be retrieved based on the compressed index includes: Decompress the inverted index dictionary and check whether the string to be searched is located in the dictionary; If the string to be searched is located in the dictionary, decompress the inverted list of the inverted index, and use the inverted list to filter the sequence of storage units to be searched; If the string to be searched is not in the dictionary, the summary index is decompressed, and the sequence of storage units to be searched is filtered by the summary index.

[0015] According to the log compression and index construction method provided by the present invention, the step of filtering the storage unit sequence to be retrieved using the inverted list includes: reading the inverted list to obtain the occurrence position of the string to be retrieved and recording it as a occurrence position list; traversing each storage unit number in the storage unit sequence to be retrieved and checking whether its corresponding storage range contains any position in the occurrence position list; if not, removing the storage unit number from the sequence, otherwise retaining the storage unit number.

[0016] According to the log compression and index construction method provided by the present invention, the step of filtering the storage unit sequence to be retrieved through the digest index includes: reading the data digest set of the digest index; traversing each storage unit number in the storage unit sequence to be retrieved and obtaining the target data digest corresponding to the storage unit number; checking whether the string to be retrieved matches the target data digest; if it does not match, removing the storage unit number from the storage unit sequence to be retrieved; if it matches, retaining the storage unit number.

[0017] According to the log compression and index construction method provided by the present invention, the step of reconstructing the target log corresponding to the search string by combining the information of the original template includes: Using the information in the original template, the constant portion of the log entry is restored; Decompress the sequence of storage units to be decompressed and restore the variable portion of the log entries; Combine the constant part and the variable part to restore the original log entry, and check whether the original log entry contains the string to be searched; if it does, output the original log entry.

[0018] The present invention also provides a log compression and index building system, the system comprising: The acquisition module is used to acquire the raw logs and divide the raw logs into raw templates and raw variables based on the log parsing tool; The indexing module is used to count the frequency of occurrence of the original variable in the original log, and to divide the original variable into variables with inverted index and variables without inverted index based on the frequency of occurrence; The construction module is used to construct an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and to construct a summary index and a corresponding second storage unit based on the variables without inverted indexes; The compression module is used to compress the inverted index and the digest index to obtain a compressed index; and to compress the first storage unit and the second storage unit to obtain compressed log data.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the log compression and index construction method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log compression and index building methods described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the log compression and index building method as described above.

[0022] The log compression and indexing construction method and system provided by this invention differentiate variables into variables with and without inverted indexes based on the frequency of occurrence of the original variables, and constructs inverted indexes, summary indexes, and corresponding first and second storage units accordingly. This effectively solves the technical defects of existing technologies, such as index bloat and high storage overhead caused by low-frequency variables. During the retrieval process, the precise positioning of the inverted index and the feature filtering mechanism of the summary index, combined with block compression storage, can quickly eliminate irrelevant data and reduce invalid decompression during large-scale log retrieval. This significantly reduces retrieval latency while maintaining a high log compression rate, achieving the best balance between large-scale log storage cost and query efficiency. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is one of the flowcharts illustrating the log compression and index building method provided by this invention.

[0025] Figure 2 This is the second flowchart illustrating the log compression and index building method provided by this invention.

[0026] Figure 3 This is a schematic diagram illustrating the construction and optimization process of the log table and variable list provided by this invention.

[0027] Figure 4 This is a schematic diagram of a specific example of the log retrieval process provided by the present invention.

[0028] Figure 5This is a schematic diagram of the log compression and index building system provided by the present invention.

[0029] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] Before describing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.

[0032] Raw template: This refers to the pattern information extracted from the raw log that describes the fixed structure of the log. It includes a sequence of delimiters, constant text, and placeholders for variable positions. The raw template itself does not contain the specific values ​​of the variables; it only retains the unchanging parts of the log entries and is the basic unit for log parsing and reconstruction.

[0033] Raw variables: These are the dynamically changing parts extracted from the raw log entries, i.e., the specific values ​​corresponding to placeholders in the log template. Raw variables are the core objects of log compression, index building, and retrieval matching.

[0034] Inverted index variables refer to raw variables that appear more frequently than a preset threshold in the original logs. Because these variables occur repeatedly, building an inverted index can achieve efficient and accurate location, resulting in significant indexing benefits. This invention constructs a dictionary and an inverted list for these variables and stores them in dictionary-encoded form.

[0035] Variables without inverted indexes: These are raw variables that appear in the original log at a frequency lower than or equal to a preset threshold, typically variables that appear only once. To avoid space expansion caused by inverted indexes, this invention does not build inverted indexes for these variables, but instead builds a lightweight summary index and stores it in its original form.

[0036] Summary Index: A lightweight index structure built for variables without inverted indexes. It generates a data summary set by extracting data features (such as numerical range, length range, required character set, and possible character set) from a fixed-length segmented variable list. The summary index does not support precise location positioning, but it can quickly exclude data blocks that do not meet the features during retrieval, reducing invalid decompression.

[0037] Inverted index: An index structure built for variables with inverted indexes, consisting of a dictionary and an inverted list. The dictionary maps variable values ​​to integer numbers in lexicographical order; the inverted list records all occurrences of that value in the original log. Inverted indexes support precise and fast location of high-frequency variables.

[0038] The first storage unit corresponds to the storage structure with inverted index variables and is organized in the form of a log table. The log table replaces variable values ​​with dictionary integer numbers and is divided into blocks according to a preset length. It supports independent compression and decompression and is a compressed storage medium for frequently used variables.

[0039] The second storage unit corresponds to the storage structure for variables without inverted indexes, organized as a variable list. The variable list preserves the original variable text, is arranged in the order of appearance and cut into fixed lengths, supports independent compression and decompression, and is a compressed storage medium for low-frequency variables.

[0040] Search string: The query string entered by the user when initiating a search, which may match the constant or variable parts of the log. During the search process, the search string goes through stages such as constant matching, index filtering, storage unit decompression, and log reconstruction, and finally locates the target log entry.

[0041] Log table: A structured table used to store inverted index variables, converted from the split original log list through dictionary encoding. In the log table, inverted index variables are replaced with integer numbers, and variables without inverted indexes are replaced with placeholders. It supports optimization techniques such as differential encoding and variable-length encoding, and is the basic unit for high-compression storage.

[0042] Original log entry: refers to a single, uncompressed log entry, consisting of constant and variable parts. During the retrieval and reconstruction phase, the decompressed storage unit is combined with the original template to reconstruct the original log entry. After secondary verification, the final retrieval result is output.

[0043] The physical execution entity of the method in this invention embodiment is an electronic device or a computer device. This device possesses data processing, storage, and communication capabilities, including but not limited to servers, workstations, cloud hosts, edge computing nodes, or dedicated log processing machines. The device internally includes at least one processor, memory, and a communication interface. The processor executes the log compression, index construction, and retrieval reconstruction steps described in this invention by running a computer program stored in the memory.

[0044] Furthermore, the software logic entity executing the method of this invention is a log processing system or log processing device. This system is deployed on the aforementioned electronic device and runs as an application, system service, containerized microservice, or function computing instance. The system is internally divided into multiple functional modules, including an acquisition module, an indexing module, a construction module, a compression module, and a retrieval module. These modules work together to perform tasks such as log parsing, variable classification, index construction, storage compression, retrieval filtering, and log reconstruction.

[0045] Figure 1 This is one of the flowcharts illustrating the log compression and index building method provided by the present invention. The method includes: Step 101: Obtain the raw logs and divide them into raw templates and raw variables based on the log parsing tool.

[0046] During the raw log acquisition phase, the system first collects real-time log stream data from various server clusters or cloud platforms. This log data is typically in plain text format and contains a large amount of common descriptive information as well as dynamic parameters that change with specific business requirements.

[0047] To achieve efficient compression and indexing, the parsing process utilizes log parsing tools to deeply split the original logs. In this step, each log entry is restored to its original template and original variables. The original template generally consists of a delimiter list, a constant section, and a variable section. The delimiter list defines the physical interval logic of log entries, while the constant section represents the semantic description in the log that does not change with specific operations; these two parts are directly considered components of the template itself.

[0048] Meanwhile, the specific values ​​of the variables are extracted from the original log text and stored independently as the original variable part. This processing method transforms the originally messy text stream into a mapping relationship composed of a stable template and a dynamic variable sequence. The reuse of the constant part and the delimiter list greatly reduces redundant data storage, while the extraction of the original variable part locks in the core business information in the log. This preprocessing mode, which completely decouples the skeleton from the content, lays a precise foundation for subsequent statistical analysis of variable distribution characteristics and the implementation of differentiated indexing strategies based on frequency of occurrence. Through this parsing mechanism, the system can accurately perceive the structural composition of the log, thereby achieving simultaneous optimization of storage efficiency and retrieval performance while ensuring data integrity.

[0049] Step 102: Calculate the frequency of occurrence of the original variable in the original log, and divide the original variable into variables with inverted index and variables without inverted index according to the frequency of occurrence.

[0050] After decomposing the original template and variables, the system enters the variable feature analysis phase. This process involves scanning all original variables and counting the frequency of each variable value at a specific template position. To ensure statistical accuracy, the system can maintain an efficient, memory-based frequency statistics table that records each variable value and its corresponding cumulative occurrence count.

[0051] Based on the frequency data obtained from statistics, the system implements a variable classification strategy. The original variables are divided into variables with inverted indexes and variables without inverted indexes. For example, in one implementation, original variables with a frequency greater than a certain frequency threshold are classified as variables with inverted indexes; original variables with a frequency less than or equal to the frequency threshold are classified as variables without inverted indexes.

[0052] Variables that appear repeatedly in the global index space are classified as variables with inverted indexes, which have high retrieval and reuse value; variables that appear very infrequently, especially unique variables that appear only once in the entire dataset, are classified as variables without inverted indexes.

[0053] This classification mechanism directly addresses the index bloat problem commonly found in log processing. By identifying low-frequency variables that do not exhibit repetition, the system avoids creating redundant dictionary entries and inverted index records for massive numbers of unique variables. This differentiated approach ensures that index resources are concentrated on high-frequency data that offers retrieval acceleration, while adopting a more compact storage and indexing scheme for low-frequency data. This controls the growth rate of index files at the source, significantly reducing the overall storage system's space overhead and providing a logical basis for building an efficient and compact index structure.

[0054] Step 103: Construct an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and construct a summary index and a corresponding second storage unit based on the variables without inverted indexes.

[0055] After classifying the variables, the system implements differentiated indexing and storage construction strategies based on the characteristics of the two types of variables. For variables with inverted indexes, the system focuses on optimizing retrieval efficiency and compressing duplicate data. The construction process includes building an inverted index for these variables, consisting of a dictionary and an inverted list. The dictionary maps the specific values ​​of the variable to compact integer numbers, while the inverted list records the precise location of these values ​​in the log. Simultaneously, the system constructs a first storage unit to store these variables, typically in the form of a log table, replacing the original strings with dictionary numbers for encoding and storage, thereby significantly reducing the storage footprint of high-frequency variables.

[0056] For variables without inverted indexes, the core objective of the system is to suppress index size growth while ensuring basic filtering capabilities. Because these variables have extremely low repetition rates, traditional inverted indexes incur significant metadata overhead, leading to a substantial decrease in index storage efficiency. Therefore, the system constructs a summary index by extracting structured features from variable fragments, such as recording the value range of values ​​or the compositional features of characters, instead of maintaining a detailed list of positions. For the persistence of these variables, the system constructs a second storage unit, organizing them in the variable list as raw text. This approach avoids the additional overhead of maintaining a dictionary for non-repeating data, ensuring the simplicity of storing low-frequency data.

[0057] This dual-track construction mechanism achieves synergistic optimization of indexing performance and storage costs. The first storage unit fully leverages the compression potential of high-frequency data through dictionary encoding, while the second storage unit avoids ineffective growth of the index space by preserving the original text and combining it with feature summaries. Through this targeted indexing and storage design, the system can allocate computing and storage resources according to the actual distribution patterns of log data, providing a highly adaptable data carrier for subsequent high-density compression and low-latency retrieval.

[0058] Step 104: Compress the inverted index and the digest index to obtain a compressed index; compress the first storage unit and the second storage unit to obtain compressed log data.

[0059] After completing the initial construction of the index and storage units, the system enters the final compression stage, aiming to minimize storage overhead through high-density encoding transformation. For the inverted index, the system implements targeted compression based on the distribution characteristics of its internal dictionary and inverted list. Since the inverted list records a large amount of log position information, and these position values ​​usually have a monotonically increasing characteristic, differential encoding is used to record only the difference between adjacent positions, thereby significantly reducing the bit width of the values. For the dictionary part, since the variable values ​​are arranged in lexicographical order and have high prefix similarity, processing with a general compression algorithm can further reduce the dictionary size.

[0060] Meanwhile, the system employs a block compression strategy for both the first and second storage units to ensure a balance between decompression efficiency and compression ratio. The first storage unit stores integer sequences converted using dictionary encoding, resulting in relatively low data entropy, suitable for integer-level compression. The second storage unit retains the original text without indexed variables, appearing as long string sequences, which the system compresses using a general compression engine. During compression, the log table and variable list are divided into independent data blocks of preset length, each compressed independently. This mechanism ensures that in subsequent retrieval stages, the system only needs to perform partial decompression on the specific storage unit that was hit, without loading the full data, thus effectively reducing computational resource consumption and decompression latency.

[0061] By compressing the index and storage units separately, the system ultimately produces compressed indexes and compressed log data. This dual compression architecture not only leverages the compression potential of structured indexes for location information but also utilizes the ability of general compression algorithms to eliminate redundant text. The resulting storage file has a compact physical layout, meeting the long-term retention requirements of large-scale log libraries on cloud or local storage media, while providing a solid technical guarantee for achieving low-cost storage and second-level retrieval response.

[0062] The log compression and index construction method provided in this invention differentiates variables into those with and without inverted indexes based on the frequency of occurrence of the original variables. It then constructs inverted indexes, summary indexes, and corresponding first and second storage units accordingly. This effectively solves the technical defects of existing technologies, such as index bloat and high storage overhead caused by low-frequency variables. During the retrieval process, the precise positioning of the inverted index and the feature filtering mechanism of the summary index, combined with block compression storage, enable the rapid elimination of irrelevant data and reduction of invalid decompression during large-scale log retrieval. This significantly reduces retrieval latency while maintaining a high log compression rate, achieving the optimal balance between large-scale log storage costs and query efficiency.

[0063] See Figure 2 The method further includes a log retrieval step, specifically including: 201. Receive a log retrieval request containing the string to be retrieved.

[0064] At the beginning of the retrieval process, the system receives a log retrieval request containing the string to be searched via an interface or interactive terminal. This request represents the user's query intent and is typically composed of a specific string of text characters, such as a specific error code, user identifier, process name, or transaction serial number. The retrieval module performs preliminary parsing of the request, extracts the core search keywords, and may combine filtering parameters such as time range and log source to limit the search space. As the triggering mechanism for the retrieval engine, this step ensures that the query command can be accurately captured and transformed into input conditions for the background index search, providing a logical starting point for efficient location across massive compressed data.

[0065] 202. Based on the information of the original template and the log retrieval request, perform constant matching on the string to be retrieved to obtain the sequence of storage units to be retrieved.

[0066] After obtaining the search string, the search engine first performs constant matching using the stored original template information. The system scans the constant portion and delimiter list in the original template, attempting to compare the search string with the fixed descriptive content in the template. If the search string overlaps with the constant portion of a template, it indicates that the keyword is highly likely to appear in log entries belonging to that template. Through this structured spatial localization, the system can quickly eliminate log categories that do not match the search string structurally at all, and based on the distribution pattern of templates in physical storage, pinpoint log blocks that may contain the target data, thereby obtaining the sequence of storage units to be searched. This process significantly narrows the search scope, providing a precise candidate set for the subsequent refined variable feature filtering stage.

[0067] 203. Based on the compressed index, filter the storage unit sequence to be retrieved to obtain the storage unit sequence to be decompressed.

[0068] After narrowing down the candidate range, the system calls the compressed index to perform further refined filtering to obtain the sequence of storage units to be decompressed. This step employs a dual-track filtering mechanism: first, the inverted index dictionary is decompressed and the search string is checked for existence. If the dictionary is matched, the precise position offset is obtained directly from the inverted list, thereby filtering out storage units that definitely contain the keyword. If the dictionary is not matched, the system switches to the summary index filtering logic, decompresses the data summary without inverted index variables, and compares whether the search string conforms to the numerical range or character value set characteristics of the corresponding storage block record. By using logical judgment at the index level instead of brute-force search, the system can eliminate a large number of storage units that, although the template matches, do not meet the variable content requirements, reducing subsequent decompression operations and significantly improving the overall throughput efficiency of the retrieval.

[0069] 204. Decompress the storage unit sequence to be decompressed, and reconstruct the target log corresponding to the search string by combining the information of the original template.

[0070] After filtering at the index level, the system performs decompression operations only on the sequence of storage units to be decompressed. In this step, the dictionary numbers in the first storage unit and the original variable text in the second storage unit are restored and logically combined with the constant part and delimiter list in the original template. The system fills in the restored variable values ​​sequentially according to the variable slots reserved in the template, thereby reconstructing the target log entry. After reconstruction, the search engine performs a final verification on the restored string to ensure that its content accurately contains the string to be searched, and returns it as the search result.

[0071] This on-demand decompression and dynamic reconstruction mode avoids invalid reading and writing of the full compressed data. While ensuring the absolute accuracy of the retrieval results, it achieves extremely low response latency and effectively solves the contradiction between long-term retention of massive logs and fast retrieval.

[0072] In an illustrative embodiment of the present invention, the specific process of filtering the storage unit sequence to be retrieved based on the compressed index is as follows: First, the search engine decompresses the inverted index dictionary corresponding to the position to be searched, which contains inverted index variables, and checks whether the user-input search string exists in the dictionary. If the search string exists in the dictionary, it further decompresses the inverted list corresponding to the search string and uses the inverted list to perform precise filtering on the current sequence of storage units to be searched; if the search string does not exist in the dictionary, it then decompresses the summary index without inverted index variables and uses the summary index to perform feature filtering on the sequence of storage units to be searched.

[0073] For scenarios using inverted lists for filtering, the search engine first reads the decompressed inverted list to obtain all occurrences of the string to be searched in the original log, and organizes these occurrences into a list of occurrences. Then, it iterates through each storage unit number in the sequence of storage units to be searched, checking whether the log range covered by that storage unit contains any occurrence from the list of occurrences. If the storage unit range does not contain any occurrences of the string to be searched, the storage unit number is removed from the sequence; if it contains at least one occurrence, the storage unit number is retained for subsequent decompression and reconstruction.

[0074] For filtering using a summary index, the search engine first reads the decompressed summary index, which is organized as a set of data summaries, with each data summary corresponding to a storage unit containing a list of segmented variables. The search engine traverses each storage unit number in the sequence of storage units to be searched, locating the corresponding target data summary based on the number. Then, it checks whether the string to be searched conforms to the feature constraints described by the data summary. For purely numeric summaries, it checks whether the value of the string is within the minimum and maximum range of the record; for character summaries, it checks whether the length of the string is within the length range of the record, and further verifies whether its character composition meets the requirements of mandatory and possible character sets. If the string does not conform to any feature of the data summary, it is determined that it cannot appear in that storage unit, and the storage unit number is removed from the search sequence; if the string conforms to all feature constraints, the storage unit number is retained for subsequent decompression and reconstruction.

[0075] Through the above filtering process, the search engine can significantly reduce the number of storage units that need to be decompressed without decompressing the complete storage units, thanks to the precise positioning capability of the inverted index and the feature exclusion capability of the summary index. This significantly reduces the I / O overhead and decompression computation overhead during the search process.

[0076] In a specific embodiment, the specific process of reconstructing the target log corresponding to the retrieval string by combining the information of the original template in step 204 is as follows: First, the search engine reads the corresponding original template information from the template library based on the original template number determined in the constant matching stage. The original template completely records the fixed structure of this type of log entry, including the sequence of delimiters, the sequence of constant words, and the relative positions of each variable in the log entry. Based on this structural information, the search engine assembles the delimiters and constant words in their original order to form the framework of the constant part of the log entry, and reserves placeholders at each variable position for subsequent variable value filling.

[0077] Secondly, the search engine performs decompression on the sequence of storage units to be decompressed after index filtering. The storage units to be decompressed are divided into two categories: For the first storage unit corresponding to an inverted index variable, its storage format is a log table, recording integer numbers encoded in a dictionary; the search engine first loads the corresponding dictionary based on the variable position of the storage unit, restoring the integer numbers one by one to the original variable strings. For the second storage unit corresponding to a variable without an inverted index variable, its storage format is a variable list, directly saving the variable values ​​in their original form; the search engine can obtain the original variable text by reading them sequentially. During decompression, the search engine strictly follows the order in which log entries appear in the original log, restoring the variable value sequence of all log entries contained in the storage unit one by one.

[0078] Subsequently, the search engine combines the restored constant and variable portions. Specifically, based on the delimiter sequence and constant word sequence recorded in the original template, the corresponding restored variable value is inserted at each variable position, and the complete log entry string is assembled according to the original word order and format of the log entry. This assembly process strictly follows the original log writing format to ensure that the reconstructed log entry is completely consistent with the original log in content and structure.

[0079] Finally, the search engine performs substring matching on the reconstructed complete log entries to check if the log entry actually contains the string to be searched entered by the user. If it does, the original log entry is output as the final search result; if it does not, the entry is discarded, and the process continues to process the next log entry in the sequence of uncompressed storage units.

[0080] Through the above reconstruction and verification process, accurate restoration from the compressed storage unit to the original target log text can be achieved.

[0081] Furthermore, the construction of the inverted index in this embodiment includes the generation of a dictionary and an inverted list. The system first extracts all values ​​of variables with inverted indices to form a non-repeating set of values, and then constructs a dictionary based on this set. When constructing the dictionary, the system maps each variable value to a unique integer number, and this mapping process follows lexicographical order rules, mapping values ​​with lower lexicographical orders to smaller integer numbers, thereby leveraging order to accelerate the subsequent dictionary lookup process.

[0082] Meanwhile, for each specific value of the inverted index variable, the system records its position information in the original log. By scanning the original log entries, the system obtains all occurrence points corresponding to each value and organizes these points into an ordered sequence, thus forming the corresponding inverted list. By associating and storing the integer numbers in the dictionary with their corresponding inverted lists, the system completes the construction of the inverted index for the inverted index variable, providing data support for quickly locating the log position based on the variable value.

[0083] Furthermore, in this embodiment, during the construction of the summary index, the system first organizes the extracted variables without inverted indexes according to their sequential appearance in the original logs, arranging them into a complete variable list. To achieve finer-grained index management and retrieval filtering, the system performs fixed-length segmentation of the variable list according to a preset length, thereby obtaining a set of segmented variable lists composed of multiple independent blocks.

[0084] For each variable list in the segmented variable list set, the system extracts data features by analyzing the specific content of the variables and constructs a corresponding data summary accordingly. During feature extraction, the system first identifies the type of the variable list, classifying it into purely numeric variable lists and character variable lists. For purely numeric variable lists, the system counts and records the minimum and maximum values ​​of all values ​​in the list, using this range of numbers as its data summary.

[0085] For lists of character variables, the system records the length range of each variable and the set of character values ​​as its data summary. The character value set is further subdivided into two parts: one part records the set of characters that each variable in the list must contain, and the other part records the set of characters that each variable in the list may contain. In this way, the system generates a concise description representing the content characteristics of each segmented variable list. Finally, the system combines these independent data summaries to obtain a complete data summary set, which is used as a summary index for variables without inverted indexes. This index is used to quickly exclude storage units that do not meet the feature requirements during retrieval.

[0086] In this embodiment of the invention, when constructing storage units, the system adopts differentiated storage strategies based on the classification results of variables to achieve a balance between high-density compression and structured restoration. For variables with inverted indexes, the system converts them into dictionary-encoded form and stores them in a log table, using this log table to construct the first storage unit. For variables without inverted indexes, the system retains their original text content and stores it in a variable list, using this variable list to construct the second storage unit. This approach achieves decoupled storage of high-frequency repetitive data and low-frequency unique data at the physical level.

[0087] During the construction of the log table, the system first arranges all variable parts of the original log entries according to their order of appearance in the original text, forming an original log list. To facilitate subsequent random retrieval and block compression, the system performs fixed-length segmentation of the original log list according to a preset number of rows or entry length, thus obtaining a set of segmented original log lists composed of multiple fragments. This block processing mechanism provides the foundation for subsequent partial decompression.

[0088] For each segmented original log list, the system performs precise replacements on its included variables. If a variable has an inverted index, the system consults a pre-built dictionary and replaces its specific value with the corresponding integer number, thus using fixed-length integer values ​​to replace variable-length strings, significantly reducing the data volume. If a variable does not have an inverted index, the system replaces it with a pre-defined placeholder. This placeholder not only marks the physical location of the variable but also ensures that the log table maintains strict consistency with the original log structure. Finally, the system organizes these replaced lists into a log table, which serves as the core carrier of the first storage unit, while the original text of the unreplaced variables without inverted indexes is stored sequentially in the corresponding second storage unit. In this way, the system maximizes the compression of storage space for high-frequency variables while ensuring that variable location information is not lost.

[0089] In this embodiment, the system further improves storage efficiency and compression rate by performing multi-dimensional encoding optimization on the log table structure. Figure 3 The process of constructing and optimizing the log table and variable list is shown.

[0090] When constructing the first storage unit, the log table, the system records the row number information of the original log entries. Since the log row numbers have the characteristic of continuous increment, the system uses differential encoding to process the row numbers, that is, it only records the difference between the current row number and the previous row number, thereby converting the larger absolute value into a smaller incremental value, which significantly reduces the storage overhead of the row number column.

[0091] For integer numbers with inverted index variables stored in the log table, considering the significant differences in dictionary size corresponding to variables at different positions, the system uses variable-length encoding for storage. For columns with smaller dictionary sizes, a smaller byte width is used for encoding; for columns with larger dictionary sizes, a wider byte width is allocated, thereby minimizing the overall bit width of the log table while ensuring data accuracy.

[0092] In addition, the system identifies and extracts variable repetition patterns in each log table block formed by the segmentation. If a column in the log table has the same value in the current data block, the specific value and its repetition pattern are recorded in the header information of the log table, and the data in that column is not recorded again in the log table body.

[0093] When processing variables, the system replaces variables with inverted indexes with their corresponding dictionary numbers, while variables without inverted indexes are replaced with preset placeholders (such as number 0). The original text of the corresponding variables without inverted indexes is then moved sequentially to the variable list for storage.

[0094] Finally, the log table and variable list, optimized through differential coding, variable-length coding, and repetition pattern extraction, are processed using a general compression algorithm to generate compressed log tables and compressed variable lists. By introducing these fine-grained coding optimizations when constructing storage units, the system can more fully exploit the structural redundancy of log data, further reducing the overall storage density of indexes and log data while ensuring random retrieval performance.

[0095] Figure 4 This document demonstrates a specific example of a log retrieval process, including retrieval, filtering, and reconstruction. In this embodiment, taking the retrieval request Query as “src:237.log dest: / Log” as an example, the specific execution process is as follows: First, after receiving the search request, the search module performs constant matching on the search string using the original template information. For example... Figure 4 As shown, the search string matched the original template "compress log src:<*>dest:<*>". Through template matching, the system identified two variables in the search string: "237.log" at position #4 and " / Log" at position #6. " / Log" was identified as a variable with an inverted index, while "237.log" was identified as a variable without an inverted index.

[0096] Subsequently, the search engine enters the efficient filtering phase. The system prioritizes processing the inverted index with stronger filtering capabilities, namely the inverted index corresponding to decompression position #6, checking if " / Log" exists in the dictionary and retrieving its inverted list. After locking down the initial sequence of storage units to be searched through the inverted list, the system further processes the summary index corresponding to position #4, performing secondary filtering on the storage unit sequence by decompressing the summary index and matching the characteristics (such as character set or length) of "237.log". This process fully utilizes the synergistic effect of differentiated indexes, obtaining the final and accurate storage units to be decompressed through layers of filtering.

[0097] Finally, the system decompresses the filtered storage units, restores the specific values ​​of the variables, and logically concatenates them with the constant parts (such as fixed text like "compress log src:") in the original template "compress log src:<*>dest:<*>" to reconstruct the complete original log entry. The reconstructed entry is then output after verification. This example intuitively demonstrates the technical effect of achieving low-latency queries in large-scale compressed logs through the retrieval path of "constant matching—hybrid index filtering—block decompression and reconstruction" in this embodiment of the invention.

[0098] The log compression and index building system provided in the embodiments of the present invention is described below. The log compression and index building system described below can be referred to in correspondence with the log compression and index building method described above.

[0099] This invention provides a log compression and index building system, see [link to documentation]. Figure 5 ,include: The acquisition module 510 is used to acquire the raw log and divide the raw log into raw templates and raw variables based on the log parsing tool; Index module 520 is used to count the frequency of occurrence of the original variable in the original log, and to divide the original variable into variables with inverted index and variables without inverted index according to the frequency of occurrence; Construction module 530 is used to construct an inverted index and a corresponding first storage unit based on the variable with inverted index, and to construct a summary index and a corresponding second storage unit based on the variable without inverted index; Compression module 540 is used to compress the inverted index and the digest index to obtain a compressed index; and to compress the first storage unit and the second storage unit to obtain compressed log data.

[0100] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a log compression and index construction method. This method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; calculating the frequency of occurrence of the raw variables in the raw logs and dividing the raw variables into variables with inverted indexes and variables without inverted indexes based on the frequency of occurrence; constructing an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and constructing a digest index and a corresponding second storage unit based on the variables without inverted indexes; compressing the inverted index and the digest index to obtain a compressed index; and compressing the first storage unit and the second storage unit to obtain compressed log data.

[0101] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log compression and index construction methods provided by the above methods. The method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; counting the frequency of occurrence of the raw variables in the raw logs and dividing the raw variables into variables with inverted indexes and variables without inverted indexes based on the frequency of occurrence; constructing an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and constructing a digest index and a corresponding second storage unit based on the variables without inverted indexes; compressing the inverted index and the digest index to obtain a compressed index; and compressing the first storage unit and the second storage unit to obtain compressed log data.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the log compression and index construction methods provided by the above methods. The method includes: acquiring raw logs and dividing the raw logs into raw templates and raw variables based on a log parsing tool; calculating the frequency of occurrence of the raw variables in the raw logs and dividing the raw variables into variables with inverted indexes and variables without inverted indexes based on the frequency of occurrence; constructing an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and constructing a digest index and a corresponding second storage unit based on the variables without inverted indexes; compressing the inverted index and the digest index to obtain a compressed index; and compressing the first storage unit and the second storage unit to obtain compressed log data.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for log compression and index construction, characterized in that, The method includes: Obtain the raw logs and divide them into raw templates and raw variables based on the log parsing tool; The frequency of occurrence of the original variables in the original log is counted, and the original variables are divided into variables with inverted indexes and variables without inverted indexes based on the frequency of occurrence. An inverted index and its corresponding first storage unit are constructed based on the variables with inverted indexes, and a summary index and its corresponding second storage unit are constructed based on the variables without inverted indexes. The inverted index and the digest index are compressed to obtain a compressed index; the first storage unit and the second storage unit are compressed to obtain compressed log data.

2. The method according to claim 1, characterized in that, The method also includes a log retrieval step: Receive log retrieval requests containing the string to be retrieved; Based on the information of the original template and the log retrieval request, constant matching is performed on the retrieval string to obtain the sequence of storage units to be retrieved; Based on the compressed index, the sequence of storage units to be retrieved is filtered to obtain the sequence of storage units to be decompressed; The sequence of storage units to be decompressed is decompressed, and the target log corresponding to the string to be retrieved is reconstructed by combining the information of the original template.

3. The method according to claim 1, characterized in that, The step of dividing the original variables into variables with inverted indexes and variables without inverted indexes based on their occurrence frequency includes: The original variables whose frequency of occurrence is greater than the frequency threshold are classified as variables with inverted indexes; The original variables whose frequency is less than or equal to the frequency threshold are classified as the variables without inverted indexes.

4. The method according to claim 1, characterized in that, The inverted index includes a dictionary and an inverted list, and its construction method includes: The dictionary is constructed based on the set of values ​​of the inverted indexed variables, and the variable values ​​are mapped to integer numbers; wherein, the mapping maps values ​​with lower lexicographical order to smaller integer numbers; For each value of the variable with inverted index, its position information in the original log is recorded to form the inverted list.

5. The method according to claim 1, characterized in that, The method for constructing the summary index includes: Arrange the variables without inverted indexes into a variable list according to their positions in the original log; The variable list is cut into fixed lengths according to a preset length to obtain a set of cut variable lists; Extract the data features of each variable list in the segmented variable list set to construct a data summary; The data digests are combined to obtain a data digest set, and the data digest set is used as the digest index.

6. The method according to claim 5, characterized in that, The step of extracting data features from each variable list in the segmented variable list set and constructing a data summary includes: Identify the type of the segmented variable list and classify it into a pure numeric variable list and a character variable list; For the list of purely numeric variables, the range of numbers is recorded as its data summary; For the list of character variables, the variable length range and the set of character values ​​are recorded as its data summary; wherein, the set of character values ​​includes a set of required characters and a set of possible characters.

7. The method according to claim 1, characterized in that, The method for constructing the first storage unit and the second storage unit includes: For the variables with inverted indexes, they are stored in a log table in dictionary form, and the log table is used to construct the first storage unit; For the variables without inverted indexes, store them in plain text in a variable list, and use the variable list to construct the second storage unit.

8. The method according to claim 7, characterized in that, The method for constructing the log table includes: Arrange the variable portions of the original log in order to form an original log list; The original log list is cut into segments according to a preset length to form a set of segmented original log lists. For each list in the set of original log lists after the splitting, replace the variable entries that belong to the inverted index variable with the integer number of the variable in the dictionary, and replace the variable entries that belong to the non-inverted index variable with placeholders; The original log list after replacement and splitting is used as the log table.

9. The method according to claim 2, characterized in that, The filtering of the storage unit sequence to be retrieved based on the compressed index includes: Decompress the inverted index dictionary and check whether the string to be searched is located in the dictionary; If the string to be searched is located in the dictionary, decompress the inverted list of the inverted index, and use the inverted list to filter the sequence of storage units to be searched; If the string to be searched is not in the dictionary, the summary index is decompressed, and the sequence of storage units to be searched is filtered by the summary index.

10. The method according to claim 9, characterized in that, The step of filtering the sequence of storage units to be retrieved using the inverted list includes: Read the inverted list to obtain the occurrence positions of the string to be searched, and record them as a occurrence position list; Traverse each storage unit number in the sequence of storage units to be retrieved, and check whether its corresponding storage range includes any position in the occurrence position list; if not, remove the storage unit number from the sequence, otherwise retain the storage unit number.

11. The method according to claim 9, characterized in that, The filtering of the storage unit sequence to be retrieved using the summary index includes: Read the data digest set of the digest index; Traverse each storage unit number in the sequence of storage units to be retrieved, and obtain the target data digest corresponding to that storage unit number; Check whether the string to be searched matches the target data digest; if it does not match, remove the storage unit number from the sequence of storage units to be searched; if it matches, retain the storage unit number.

12. The method according to claim 2, characterized in that, The process of reconstructing the target log corresponding to the search string by combining the information from the original template includes: Using the information in the original template, the constant portion of the log entry is restored; Decompress the sequence of storage units to be decompressed and restore the variable portion of the log entries; Combine the constant part and the variable part to restore the original log entry, and check whether the original log entry contains the string to be searched; if it does, output the original log entry.

13. A log compression and index building system, characterized in that, The system includes: The acquisition module is used to acquire the raw logs and divide the raw logs into raw templates and raw variables based on the log parsing tool; The indexing module is used to count the frequency of occurrence of the original variable in the original log, and to divide the original variable into variables with inverted index and variables without inverted index based on the frequency of occurrence; The construction module is used to construct an inverted index and a corresponding first storage unit based on the variables with inverted indexes, and to construct a summary index and a corresponding second storage unit based on the variables without inverted indexes; The compression module is used to compress the inverted index and the digest index to obtain a compressed index; and to compress the first storage unit and the second storage unit to obtain compressed log data.

Images