Log processing method and system

Abstract

This invention relates to the field of data processing technology, providing a log processing method and system. The method includes: recording the location information of sub-variables that need to be compressed in a log block as key positions and corresponding metadata of the encoded stream into a preset data structure based on a preset template; compressing the log block into different storage units based on the preset data structure; locating the storage unit corresponding to the search keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and completing the retrieval of log entries based on the matching result of the search keyword and the storage unit, where the location vector is the matching result of the search keyword and the decompressed preset template. This invention efficiently extracts dynamic patterns from logs through preset templates and stores log data as fine-grained units, thereby improving log compression bandwidth; it uses vectorization technology to achieve low-latency retrieval of fine-grained units, thus providing effective support for log analysis and system modeling.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log processing method and system. Background Technology

[0002] Currently, large cloud service providers generate massive amounts of system logs daily, sometimes reaching petabyte (PB) levels. These logs record critical system information, capture user behavior, and are crucial for error diagnosis, modeling, and security intrusion detection. Due to the sheer volume of this data, domain-specific log compression algorithms are typically used. Furthermore, given the rapid rate of log data generation, high-speed compression must be achieved while writing logs quickly. To support these applications, log retrieval needs to be performed with low latency.

[0003] Existing log storage systems typically store raw log data into fine-grained units by dynamically extracting log templates to reduce decompression overhead. Simultaneously, to achieve fast writes, existing systems usually extract log templates from log samples first. However, these pre-trained templates can lead to increased mismatch rates in practical use, thus affecting compression ratios and increasing retrieval latency. Therefore, improving log write bandwidth while maintaining compression ratios and reducing retrieval latency has become a pressing issue. Summary of the Invention

[0004] This invention provides a log processing method and system to solve the defects in the prior art where, when storing raw log data into fine-grained units using a dynamic log template extraction method, the log template needs to be extracted from the log sample first, which causes an increase in mismatch rate, thereby affecting the compression rate and increasing retrieval latency.

[0005] This invention provides a log processing method, comprising:

[0006] Based on a preset template, the position information of the sub-variables that need to participate in compression in the log block is recorded as key points and metadata corresponding to the encoded stream in a preset data structure. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template.

[0007] Based on the preset data structure, the log blocks are compressed into different storage units;

[0008] Based on the distribution of the location vectors corresponding to each log block on the location matrix, the storage unit corresponding to the search keyword is located, and based on the matching result between the search keyword and the located storage unit, the retrieval of log entries in each log block is completed based on the search keyword. The location vector is the matching result between the search keyword and the decompressed preset template.

[0009] The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the local template pattern is filled with the variable values ​​extracted from the variable intervals corresponding to the local template pattern.

[0010] According to the log processing method of the present invention, the global template is the template corresponding to the template with the largest number of templates among several template patterns obtained by training the variable group; the local template patterns are all other template patterns except the template pattern corresponding to the global template.

[0011] The constant information and the type and length information are obtained based on the pattern of the global template and the variable group during training.

[0012] According to the log processing method of the present invention, the preset data structure is a data structure that utilizes the prefetch function of the CPU cache for acceleration.

[0013] The location information of the sub-variables in the log block that need to participate in compression is recorded as key positions and corresponding metadata of the encoded stream into a preset data structure, including:

[0014] The offset and length of each key point are stored in an entry of a preset interleaving table. Each row of the preset interleaving table is aligned with the CPU cache row of the preset data structure. Each row records multiple key points belonging to the same encoding stream and a pointer to the next row belonging to the same encoding stream.

[0015] A contiguous array of metadata stores metadata corresponding to different encoded streams belonging to the same static template.

[0016] According to the log processing method of the present invention, the step of compressing the logs into different storage units based on the preset data structure includes:

[0017] Based on the preset data structure, the position information of the corresponding sub-variables contained in the global template and the local template style are extracted respectively;

[0018] Based on the position information of the sub-variables extracted from the global template, the sub-variables of the same type are stored consecutively in one storage unit;

[0019] Based on the position information of the sub-variables extracted from the same local template, the sub-variables are filled with a preset length and stored continuously in the same storage unit.

[0020] According to the log processing method of the present invention, the step of locating the storage unit corresponding to the search keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and completing the retrieval of log entries in each log block based on the search keyword based on the matching result of the search keyword and the located storage unit, includes:

[0021] Obtain search keywords;

[0022] Based on the search keywords, match the decompressed preset template;

[0023] The matching results are stored as position vectors in the position matrix;

[0024] Based on the distribution of the location vectors corresponding to each log block on the location matrix, the location vectors are merged and / or pruned to locate the storage unit corresponding to the search keyword;

[0025] Based on the search keywords, the storage units located are matched to obtain intermediate search results corresponding to each log block;

[0026] The intermediate search results are merged and / or pruned to obtain the final search results, thus completing the search for log entries in each log block based on the search keywords.

[0027] According to the log processing method of the present invention, the step of matching and locating the storage unit based on the search keyword to obtain the intermediate search results corresponding to each log block includes:

[0028] Based on the key parameters and the search keywords, unnecessary storage units are filtered out. The unnecessary storage units are those that do not contain the search keywords.

[0029] The remaining storage units after filtering are decompressed, and when searching using an integer matching algorithm, the potential occurrence area of ​​the target content is determined based on the key parameters and the search keywords. The target content is the log entry stored in the storage unit that corresponds to the search keywords.

[0030] Based on the search keywords, the target content is searched in the potential occurrence area to obtain the intermediate search results;

[0031] The key parameters were determined using the following method:

[0032] Data feature analysis is performed on the variables of the log extracted by the preset template to determine the data features of the variable data, including distribution features and changing trends;

[0033] The data features are profiled to generate key parameters that reflect the characteristics of the variables.

[0034] According to the log processing method of the present invention, the step of matching and locating the storage unit based on the search keyword to obtain the intermediate search results corresponding to each log block further includes:

[0035] Determine whether the remaining storage units after filtering are of dictionary type;

[0036] After decompressing the dictionary index unit of the storage unit of the dictionary type, the text matching method is used. After a successful match, the dictionary value of the storage unit of the dictionary type is decompressed, and the integer matching algorithm is used for retrieval.

[0037] For storage units that are not of dictionary type, the data type of the storage unit is determined. If the data type is text, a string matching algorithm is used for retrieval. If the data type is integer, an integer matching algorithm is used for retrieval.

[0038] According to the log processing method of the present invention, the position matrix includes a first-level position matrix and a second-level position matrix;

[0039] The first-level position matrix is ​​used to store the first position vector, which is the matching result of the decompressed static template based on the search keyword; the first position vector includes a variable identifier marking a successful match, the split search keyword, the matching alignment of the search keyword on the static template, and the matching status;

[0040] The second-level position matrix is ​​used to store the second position vector, which is the matching result of the decompressed dynamic template based on the search keyword; the second position vector includes the variable identifier of the successful match, the split search keyword, and the matching alignment of the search keyword on the dynamic template;

[0041] The static template is first decompressed and matched. If the matching status in the first position vector is successful, the search result is returned directly. Otherwise, the dynamic template is decompressed and matched.

[0042] According to the log processing method of the present invention, merging and / or pruning the intermediate retrieval results to obtain the final retrieval results includes:

[0043] The intermediate retrieval results are merged and / or pruned based on a vectorized bitmap structure.

[0044] The merged and / or pruned intermediate retrieval results are stored in the repository of the vectorized bitmap structure;

[0045] Based on the data in the warehouse, the logs are restructured to obtain the final search results.

[0046] The present invention also provides a log processing system, comprising:

[0047] The writing module is used to record the position information of the sub-variables that need to be compressed in the log block as key points and metadata corresponding to the encoded stream into a preset data structure based on a preset template. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template.

[0048] A compression module is used to compress the log blocks into different storage units based on the preset data structure;

[0049] The retrieval module is used to locate the storage unit corresponding to the retrieval keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and to complete the retrieval of log entries in each log block based on the retrieval keyword based on the matching result of the retrieval keyword and the located storage unit. The location vector is the matching result of the retrieval keyword and the decompressed preset template.

[0050] The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into a local template based on the variable values ​​extracted from the variable intervals corresponding to the local template pattern.

[0051] The log processing method and system provided by this invention record the location information of the sub-variables that need to be compressed in the log block as key positions and the corresponding metadata of the encoded stream in a preset data structure through a preset template. This allows the log blocks to be compressed into different storage units based on the preset data structure during log encoding and compression. By using the matching result of the search keyword and the decompressed preset template as a position vector, the storage unit corresponding to the search keyword is located based on the distribution of the position vectors corresponding to each log block on the position matrix. Based on the matching result of the search keyword and the located storage unit, the retrieval of log entries in each log block is completed based on the search keyword. Low-latency retrieval on the storage unit is achieved based on vectorization technology.

[0052] Meanwhile, the preset templates include static templates and dynamic templates. The dynamic templates further include global templates and local template styles, which allows key points for compression to be marked when log blocks are written. During actual compression, the local template style is determined based on the actual shape of the log block and the details are filled in. This not only minimizes copying during the log writing process but also ensures a high compression rate for the logs. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a log processing method provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the dynamic template and its dynamic adjustment provided in the embodiments of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of the interleaved table provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the process for implementing log retrieval on a two-level location template provided by an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the structure of the vectorized bitmap provided in an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the generation of intermediate retrieval results based on vectorized shuffling provided in an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of a mask-based vectorized retrieval provided in an embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of the log retrieval process using the log processing method provided in the embodiments of the present invention;

[0062] Figure 9 This is a schematic diagram illustrating vectorized retrieval based on dictionary-encoded variables, provided by an embodiment of the present invention.

[0063] Figure 10 This is a schematic diagram of vectorized prefix matching for integer encoded variables provided in an embodiment of the present invention;

[0064] Figure 11 This is a schematic diagram illustrating the process of converting prefix matching into interval matching according to an embodiment of the present invention;

[0065] Figure 12 This is a schematic diagram illustrating the matching of the same prefix in different intervals provided by an embodiment of the present invention;

[0066] Figure 13 This is a flowchart illustrating the process of recognizing integer variable change features provided in an embodiment of the present invention;

[0067] Figure 14 This is a schematic diagram of the structure of a log processing system provided in an embodiment of the present invention;

[0068] Figure 15 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0070] The following is combined with Figures 1 to 13 This invention describes a log processing method that can be executed by software and / or hardware in an electronic device such as a computer, such as... Figure 1 As shown, the method includes the following steps:

[0071] 101. Based on a preset template, the position information of the sub-variables that need to participate in compression in the log block is recorded as key points and metadata corresponding to the encoded stream in a preset data structure. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template.

[0072] 102. Based on the preset data structure, the log blocks are compressed into different storage units.

[0073] The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into a local template based on the variable values ​​extracted from the variable intervals corresponding to the local template pattern.

[0074] It is understandable that in the actual compression process, logs are usually divided into multiple log segments for compression. Therefore, the embodiments of this application compress logs in units of log segments.

[0075] Furthermore, each variable group contains multiple variables, each with its own value. The range of values ​​for all variables within a variable group is called the variable interval. In this embodiment, log samples are divided into static templates and variable groups. Training samples are then obtained by sampling the variable groups. Before writing the logs, the global template and several selectable local template styles are pre-trained using the training samples. This allows the dynamic template to be determined based on the actual variable values ​​in the log block during actual compression. If the corresponding variable value belongs to the global template, the position information of the sub-variables contained therein is directly extracted. If it corresponds to a local template style, the corresponding style information is filled and updated based on the corresponding variable value, thereby minimizing data copying during the log writing process. Meanwhile, during the log writing process, the location information of sub-variables for compression can be marked based on the global template and the local template style corresponding to the actual form of the log. At the same time, the metadata corresponding to the encoding stream includes the current filtering status and encoding amount of the independent variables included in the encoding stream. Therefore, by recording the location information of the sub-variables as key positions and the metadata corresponding to the encoding stream in the preset data structure, the relevant positions in the log can be encoded according to the preset data structure during actual compression.

[0076] In one specific embodiment Figure 2 This is a schematic diagram of a dynamic template example and its dynamic adjustment process provided by an embodiment of the present invention. By training the variable group extracted from the log sample, a global template (Global Pattern in the figure) that records the constant information and length and type information of the sub-variables in the dynamic template can be generated, as well as several selectable local template styles (Local Sketches in the figure) that only contain the number of sub-variables and the delimiter information between variables. When compressing each log block, the compression algorithm will select the local template style to participate in the compression according to the local template actually contained in the log block, and at the same time dynamically update the sub-variable type and length information in the local template according to the actual data information.

[0077] Furthermore, when training based on the variable group of log samples, several template patterns can be obtained. It is understandable that the template pattern with the largest number of occurrences best reflects the overall shape of the log samples. Moreover, training based on the variable group and the template pattern can obtain information such as the constant information and the type and length of the sub-variables contained in the template pattern.

[0078] Based on this, in one embodiment of the present invention, the global template is the template corresponding to the template with the largest number of template styles among several template styles obtained by training the variable group; the local template styles are all other template styles except the template style corresponding to the global template.

[0079] The constant information and the type and length information are obtained based on the pattern of the global template and the variable group during training.

[0080] In one embodiment of the present invention, the preset data structure is a data structure that utilizes the prefetch function of the CPU cache for acceleration.

[0081] Based on a preset template, the location information of the sub-variables in the log block that need to participate in compression is recorded as key positions and corresponding metadata of the encoded stream in a preset data structure, including:

[0082] The offset and length of each key point are stored in an entry of a preset interleaving table. Each row of the preset interleaving table is aligned with the CPU cache row of the preset data structure. Each row records multiple key points belonging to the same encoding stream and a pointer to the next row belonging to the same encoding stream.

[0083] A contiguous array of metadata stores metadata corresponding to different encoded streams belonging to the same static template.

[0084] In this embodiment, key log positions and corresponding metadata of the encoded stream are recorded in an interleaved table based on a dynamic template. During actual encoding, interleaved encoding of different encoded streams can be achieved based on the interleaved table and the metadata table.

[0085] Specifically, by using an "interleaved table" to store the offsets and lengths of key log positions that need to be encoded, each row of the interleaved table is aligned with a CPU cache line, which stores several log positions belonging to the same encoding stream, and also records the next row belonging to that encoding stream, which can further improve the log write bandwidth.

[0086] Furthermore, by using a contiguously stored metadata array, the metadata corresponding to different encoded streams belonging to the same static template is stored together contiguously. This allows all encoded streams belonging to the static template to which the log belongs to be updated simultaneously during actual compression, so that the corresponding metadata information can be updated simultaneously during the actual update process.

[0087] In one specific embodiment Figure 3This is a schematic diagram of an interleaved table structure provided in an embodiment of the present invention. Each entry in the interleaved table contains two integers: one is the offset value of the key point to be encoded, and the other is the length of the key point to be encoded. Each row of the interleaved table is the same length as a data row in the cache (typically 64MB), specifically containing the position information of 7 pairs of key points belonging to the same encoding stream and an additional integer (288 in the first row of the figure). This integer records the next interleaved table row belonging to this encoding stream. If the value is -1, it indicates that the row is the last interleaved table row of the current encoding stream. On the other hand, Figure 3 The bottom table contains metadata information for different encoding streams. The metadata for each encoding stream includes the position of the first interleaving table row (First), the position of the last interleaving table row (Last), and the type (Type) and length information (Len) of the key points corresponding to the current encoding stream.

[0088] For variable ranges belonging to the global template, the compression algorithm will extract the sub-variables contained therein and store sub-variables of the same type consecutively in a fine-grained storage unit (Capsule). For variable ranges belonging to the same local template, the compression algorithm will fill them with a fixed length and store them consecutively in the same fine-grained storage unit.

[0089] In one embodiment of the present invention, when the log block is compressed to different storage units based on the preset data structure, the location information of the sub-variables corresponding to the global template and the local template style are extracted respectively based on the preset data structure;

[0090] Based on the position information of the sub-variables corresponding to the global template, the sub-variables of the same type are stored consecutively in one storage unit;

[0091] Based on the position information of the sub-variables corresponding to the same local template, the sub-variables are filled to a preset length and stored continuously in one storage unit.

[0092] Understandably, logs record critical system information, capture user behavior, and are crucial for error diagnosis, modeling, and security intrusion detection. To support these applications, log retrieval needs to be performed with low latency. However, existing log storage systems use pre-trained log templates that can lead to increased mismatch rates in real-world applications, thus increasing retrieval latency.

[0093] Based on this, after compressing the log blocks into different storage units according to the aforementioned steps 101 and 102, the method further includes a low-latency retrieval step on these storage units based on vectorization technology, namely:

[0094] 103. Based on the distribution of the position vectors corresponding to each log block on the position matrix, locate the storage unit corresponding to the search keyword, and based on the matching result of the search keyword and the located storage unit, complete the retrieval of log entries in each log block based on the search keyword, wherein the position vector is the matching result of the search keyword and the decompressed preset template.

[0095] In one embodiment of the present invention, the retrieval process specifically includes:

[0096] Obtain search keywords;

[0097] Based on the search keywords, match the decompressed preset template;

[0098] The matching results are stored as position vectors in the position matrix;

[0099] Based on the distribution of the location vectors corresponding to each log block on the location matrix, the location vectors are merged and / or pruned to locate the storage unit corresponding to the search keyword;

[0100] Based on the search keywords, the storage units located are matched to obtain intermediate search results corresponding to each log block;

[0101] The intermediate search results are merged and / or pruned to obtain the final search results, thus completing the search for log entries in each log block based on the search keywords.

[0102] It is understandable that when matching a preset template based on search keywords, multiple variables may be matched simultaneously.

[0103] In this embodiment, when performing a search, the user-input search content, i.e., the search string, is first decomposed into several search keywords. Then, a preset template is decompressed, and the decompressed preset template is matched according to the search keywords. By storing the matching results as position vectors in a position matrix, the position vectors can be merged or pruned based on the position matrix to locate the storage unit corresponding to the search keyword. Then, based on the search keyword, the located storage unit is further matched to obtain the intermediate search results corresponding to each log block. Finally, the intermediate search results are merged and / or pruned to obtain the final search result, thus completing the search for log entries in each log block based on the search keywords.

[0104] Understandably, the retrieval process requires first decompressing the storage units, and then, after identifying the storage units containing the search keywords, further searching for content containing those keywords within those units. Therefore, reducing the number of storage units that need to be decompressed and pre-determining the possible locations of the search keywords within the storage units can further reduce retrieval latency.

[0105] Based on this, in one embodiment of the present invention, based on the search keywords, the located storage units are matched to obtain intermediate search results corresponding to each of the log blocks, including:

[0106] Based on the key parameters and the search keywords, unnecessary storage units are filtered out. The unnecessary storage units are those that do not contain the search keywords.

[0107] The remaining storage units after filtering are decompressed, and when searching using an integer matching algorithm, the potential occurrence area of ​​the target content is determined based on the key parameters and the search keywords. The target content is the log entry stored in the storage unit that corresponds to the search keywords.

[0108] Based on the search keywords, the target content is searched in the potential occurrence area to obtain the intermediate search results;

[0109] The key parameters were determined using the following method:

[0110] Data feature analysis is performed on the variables of the log extracted by the preset template to determine the data features of the variable data, including distribution features and changing trends;

[0111] The data features are profiled to generate key parameters that reflect the characteristics of the variables.

[0112] In this embodiment, by analyzing the distribution characteristics and trends of variables in the logs, and then using a feature extraction algorithm, corresponding filtering metadata is generated. By introducing metadata, the retrieval efficiency of log data can be improved from both vertical and horizontal dimensions while reducing storage costs. The vertical dimension refers to reducing the decompression overhead of storage units, ensuring that as many storage units as possible that do not contain the search keywords are filtered out in advance. The horizontal dimension refers to the ability to examine only the possible locations of search values ​​when retrieving data from storage units based on the distribution characteristics provided by the metadata, thereby reducing the search space for variables.

[0113] In one embodiment of the present invention, based on the search keywords, matching and locating the storage units to obtain intermediate search results corresponding to each of the log blocks further includes:

[0114] Determine whether the remaining storage units after filtering are of dictionary type;

[0115] After decompressing the dictionary index unit of the storage unit of the dictionary type, it is matched using text mode. After a successful match, the decompressed dictionary value of the storage unit of the dictionary type is retrieved using the integer matching algorithm.

[0116] For storage units that are not of dictionary type, the data type of the storage unit is determined. If the data type is text, a string matching algorithm is used for retrieval. If the data type is integer, an integer matching algorithm is used for retrieval.

[0117] In one embodiment of the present invention, the position matrix includes a first-level position matrix and a second-level position matrix;

[0118] The first-level position matrix is ​​used to store the first position vector, which is the matching result of the decompressed static template based on the search keyword; the first position vector includes a variable identifier marking a successful match, the split search keyword, the matching alignment of the search keyword on the static template, and the matching status;

[0119] The second-level position matrix is ​​used to store the second position vector, which is the matching result of the decompressed dynamic template based on the search keyword; the second position vector includes the variable identifier of the successful match, the split search keyword, and the matching alignment of the search keyword on the dynamic template;

[0120] The static template is first decompressed and matched. If the matching status in the first position vector is successful, the search result is returned directly. Otherwise, the dynamic template is decompressed and matched.

[0121] In this embodiment, as Figure 4 This demonstrates a two-level positional template matching strategy. A positional matrix is ​​used to record the matching results. The positional matrix has two levels. The first-level positional matrix stores the matching results for the static template. Its position vector mainly describes the successfully matched variable identifier, the split search keyword, the alignment of the search keyword on the template, and the matching status. The matching alignment is crucial to the accuracy of variable retrieval and also provides a reference for optimizing the search algorithm. If the matching status is 0 or 1, it indicates a direct match on the static global template string, and the search result can be returned directly; otherwise, further matching on the dynamic template is required. The second-level positional matrix stores the matching results for the dynamic template. Its position vector mainly consists of three elements: variable identifier, search keyword, and alignment, with the same meaning as the first-level positional matrix.

[0122] Furthermore, when matching search keywords, multiple variables may be matched simultaneously, such as... Figure 4 In <v0>and <v1>Positions are related by an "AND" relationship, which is represented as multiple positional variables in a row in the position matrix. A pruning algorithm is used during the variable matching phase. Search keywords may also be matched in multiple different positions on the template, such as... <v2>Location, at this time <v0> : <v1>and <v2>It is an "OR" relationship, which is represented as multiple rows in the position matrix. A merging algorithm is used to process this during the variable matching stage. Both levels of position matrices support multiple "AND" and "OR" matching relationships.

[0123] Furthermore, the position vector can be expanded with more elements to describe data features according to matching requirements, thus achieving strong scalability and representation capabilities while occupying less memory space.

[0124] In one embodiment of the present invention, merging and / or pruning the intermediate retrieval results to obtain the final retrieval results includes:

[0125] The intermediate retrieval results are merged and / or pruned based on a vectorized bitmap structure.

[0126] The merged and / or pruned intermediate retrieval results are stored in the repository of the vectorized bitmap structure;

[0127] Based on the data in the warehouse, the logs are restructured to obtain the final search results.

[0128] In this embodiment, by introducing a vectorized bitmap structure (VecBitmap), intermediate results of data retrieval are recorded with relatively low storage overhead, while supporting data-level parallel computing and fast storage of results.

[0129] Figure 5 The specific structure of the VecBitmap used in this embodiment of the invention is shown, such as... Figure 5 As shown, its essence is to add an inverted index table and byte alignment to the bitmap structure, thereby enabling fast location and calculation of valid positions in the bitmap using a vectorized approach. VecBitmap uses a union approach to partition the bitmap memory, enabling the bitmap to support 8-way, 16-way, and 32-way data parallelism under the Intel AVX2 instruction set.

[0130] Specifically, Figure 6 and Figure 7 This demonstrates the process of integer equality matching in VecBitmap during merging and pruning operations. When there is no index, the retrieval process uses a merging strategy, sequentially reading data from the data (Text) involved in the retrieval for matching. Figure 6 Taking 16-way SIMD (Single Instruction Multiple Data) as an example, firstly, 16 integers are loaded sequentially into vector register R1. Simultaneously, the search keyword is expanded into 16-way data and loaded into register R2. Then, R1 and R2 are compared for equality, generating a mask. The mask indicates whether each integer bit matches. The mask, along with the index array of values ​​in the Text, is rearranged according to a pre-defined shuffling rule (e.g., a constant matrix shuffle matrix). Simultaneously, the positions of '1's in the mask are shuffled to the front, and the index values ​​of the Text are also shuffled to the front accordingly.

[0131] Slightly different from the merging strategy, the pruning strategy needs to read the data at the corresponding position in the Text based on the Index value of the VecBitmap, which ensures that the vectorized matching data is valid. Figure 7 This demonstrates the process of vectorized pruning in 4-way parallel mode. First, the data in the Text is loaded into register R1 according to the Index, and the search keywords are expanded to 4-way and loaded into register R2. Then, an equality comparison is performed to generate a mask. The mask value and the 4 values ​​in the Index are shuffled to generate a new position sequence, which is then written back to the Index of the VecBitmap.

[0132] In one specific embodiment Figure 8 This paper demonstrates the specific process of log retrieval using the log processing method provided in this embodiment of the invention. The storage unit is referred to as a "box," and metadata filtering refers to the process by which each log block locates the matching range based on its own metadata information and a location matrix. Furthermore, log retrieval typically needs to support multiple retrieval modes simultaneously, such as point queries, range queries, and full-text queries. However, existing systems often face conflicts between storage formats and query modes; for example, string encoding is suitable for full-text queries but not for range queries, while integer encoding is suitable for range queries but not for full-text queries. Figure 8 As can be seen, the log processing method provided in this embodiment of the invention is based on the SIMD vectorization technology, which enables multimodal queries during retrieval. At the same time, it also uses vectorized shuffling to generate intermediate results of log queries to support subsequent merging, thus realizing extremely low-latency vectorized queries that support multimodal queries.

[0133] The following sections will specifically illustrate the log retrieval process using the log processing method provided in this embodiment of the invention, based on several commonly used modalities in current retrieval methods.

[0134] Understandably, commonly used search modalities include point search, range search, and full-text search. Among them, integer-based full-text matching is further divided into three types: "prefix matching," "suffix matching," and "matching at any position."

[0135] Point queries and range queries determine whether two integers are exactly equal or whether an integer falls within a certain range. They simultaneously perform full matches on the values ​​of dictionary-encoded variables and some non-dictionary integer variables. For dictionary variables, the search may hit multiple dictionary indices, resulting in cases of multiple-value full matches.

[0136] Prefix matching starts from the leftmost character of the search string and the target string (both strings are integers) and matches to the right, with each character being identical, until the entire search string has been traversed.

[0137] Suffix matching involves matching from the rightmost character of both the search string and the target string to the left, ensuring that every character matches the other character, until the entire search string has been traversed.

[0138] Arbitrary position matching means that the search string is a substring of the target string, and there are no requirements on the starting position of the match. The arbitrary position matching process follows the constraints of formula (1), where x represents the target variable to be searched and y represents the search string.

[0139] f(x,y)=x / 10 i mod 10 j ;ji=len(y) (1)

[0140] Specifically, Figure 9 This demonstrates the process of performing multi-value full matching using the log processing method provided in this embodiment of the invention:

[0141] Multivalued full matching only occurs when matching the canonical variables. Figure 9 The example demonstrates a dictionary variable data segment consisting of two dictionary index boxes (.dic) and one dictionary value box (.entry). The .entry stores the global index number (gId) of the .dic for that variable. If the search keyword is "fuxi", the search program first performs string matching on the two .dic files, saving the results to a temporary VecBitmap. This bitmap records whether the value of the corresponding gId row in the .dic is 1; therefore, the size of the bitmap must be at least larger than the size of the gId. When the amount of matching result data is small, multiple rounds of SIMD computation can be used to speed up the matching process. When the amount of matching result data is large, since the .entry stores the gId value of the .dic, the search result can be directly determined based on the value of the bitmap to see if it matches any .entry.

[0142] Figures 10 to 12 The process of prefix matching using the log processing method provided by the embodiments of the present invention is shown:

[0143] Among them, a squeezing comparison method based on 8 - base vectorization is adopted, that is, the m - th power of 8 (in the case of 4 - byte integers: 0 < m < 11) is used as the scale ruler to utilize fast shift operations to replace complex operations. By performing two expansions on the retrieved integer and using the results of two comparisons to determine whether the matching is successful, the specific matching process is as Figure 10 shown.

[0144] In a specific embodiment, Figure 11 the corresponding situations of the number 502 under different scale rulers are shown. It can be seen that for the 8 - base scale, there may be a certain scale interval that is straddled. For example, the index values of 502 and 5020 in the 8 - base scale are 2 and 4 respectively. This situation will occur at most once within the range of 4 - byte integers. The final expansion result is as Figure 12 shown. Since the 8 - base scale is smaller than the 10 - base scale, there may be one more matching step than the 10 - base scale in the matching process of the algorithm. However, since shift operations can be used to replace time - consuming data loading, the overall algorithm efficiency is higher.

[0145] Figure 10 What is shown is the left - aligned vectorized matching process under the 8 - base squeezing method. In the whole matching process, the variable is shifted to the right. If the variable becomes 0 after being shifted to the right by 3 bits, it means that it coincides with one of the scale values, so the retrieved value is retained; otherwise, the next - level retrieved value is loaded. Since the shift operation of the variable itself only needs to be performed in the same vector register, the matching efficiency is higher. The core idea of prefix matching is to convert string matching into a small number of more efficient integer comparisons. Therefore, the fewer the number of integer comparisons, the higher the efficiency.

[0146] When performing suffix matching using the log processing method provided by the embodiments of the present invention, first take the modulus of the integer variable, where the size of the modulus is the m - th power of 10 (m = the character length of the retrieved value), and then load the value after taking the modulus and the retrieved value into the vector register for SIMD comparison. Among them, in order to accelerate the modulus - taking process, an immediate number can be used as the value of the modulus.

[0147] When performing matching at any position using the log processing method provided by the embodiments of the present invention, based on the integer variable change feature recognition process as Figure 13 shown, the selection of the retrieval interval is performed. Among them, a quadruple (tag, pos, min, max) is used to describe the monotonic interval, and its meaning is:

[0148] tag: Identifies the type of change in the interval. A value of 0 indicates an equal interval; 1 indicates an increasing interval; 2 indicates a decreasing interval; further subdividing the interval into random fluctuation intervals, a value of 3 indicates a general random fluctuation interval, and a value of 4 indicates an outlier interval;

[0149] pos: Indicates the end position of the interval. Since the intervals are continuous, the search range for the i-th interval is [posi-1, posi];

[0150] min: the minimum value of the interval;

[0151] max: The maximum value in the interval. When tag is 4, only the maximum value is considered.

[0152] During retrieval, the range to be searched can be selected based on the maximum and minimum values. For searches outside the range, pruning can be performed to reduce the search space. In range retrieval, the identity range is the best range, as it can be directly matched based on the maximum value; the monotonic range is a relatively good search range because the data is ordered, and the search position can be quickly located using binary search; while the random fluctuation range requires traversing the entire range for retrieval.

[0153] In a specific embodiment, a three-point backtesting dynamic interval partitioning method can be further provided to dynamically partition and merge intervals, namely:

[0154] First, three main marker points are set: Prepre represents the starting position of the interval to be divided; Pre represents the position to be split or merged; and Cur represents the position of the point where monotonicity changes or the transition value is too large (outlier point). Four threshold values ​​are also set: minR represents the minimum range that the monotonic interval can be divided into; maxR represents the maximum range that the fluctuating interval can be divided into; width represents the normal threshold value, used to detect outliers, and its value can be dynamically adjusted according to the size of the variable; and maxU represents the threshold value for the difference between the maximum and minimum values ​​of two mergeable intervals.

[0155] It is understandable that the interval from Cur to Pre is generally a monotonically consistent interval, while the interval from Cur to Prepre is definitely not a monotonically consistent interval; each time a split or merge event is triggered, it must be the appearance of the Cur point.

[0156] Therefore, if the distance from Cur to Pre is greater than minR, then Pre and Cur form an interval. In this case, if Prepre and pre are not the same point, then Prepre and pre also need to be forced to form an interval. If the distance from Cur to Pre is less than minR, but the distance from Cur to Prepre is greater than maxR, then Cur and Prepre form an interval. If the distance from Cur to Pre is less than minR and the distance from Cur to Prepre is also less than maxR, then the point Cur becomes the point Pre. If Cur is an outlier, then the starting position of the outlier point, Cur′, is recorded separately. When Cur recovers to a normal point, Cur and Cur′ form an interval.

[0157] In addition, each time a new interval split is triggered, an attempt is made to merge it with the previous interval. The conditions for merging are: the difference between the maximum value and the difference between the minimum value of the two intervals are both less than maxU; or the range of one interval contains the range of another interval; or the interval domain is too small.

[0158] Based on the above three-point backtesting dynamic interval partitioning method, dynamic interval partitioning on the storage unit can be realized, and the representation effect has a stronger representation ability compared with the static partitioning algorithm.

[0159] In summary, the log processing method provided by the above embodiments of the present invention, on the one hand, obtains training samples by sampling log samples. Before writing the log, it uses the training samples to pre-train a global template and several selectable local template styles for the log. During actual compression, it determines the local template style based on the actual form of the log and fills in the details, thereby minimizing copying during the log writing process. Simultaneously, it marks key points for compression during writing and records the relevant information in a data structure called a "cross-tabulation." During actual compression, it interleaves-encodes the relevant points in the log according to this data structure. On the other hand, during retrieval, it supports low-latency point queries, range queries, and full-text queries based on vectorization technology. It also uses vectorized shuffling to generate intermediate log query results to support subsequent merging. Thus, it can achieve high write bandwidth while ensuring high log compression ratio and low retrieval latency, and also supports multimodal, extremely low-latency vectorized queries, providing effective support for log analysis and system modeling.

[0160] Based on the same general inventive concept, this invention also protects a log processing system. The log processing system provided by this invention will be described below, and the log processing system described below can be referred to in correspondence with the log processing method described above.

[0161] Figure 14 This is a schematic diagram of the log processing system provided by the present invention. Figure 14 As shown, it includes: a writing module 1410, a compression module 1420, and a retrieval module 1430; wherein,

[0162] The writing module 1410 is used to record the position information of the sub-variables that need to participate in compression in the log block as key points and metadata corresponding to the encoded stream into a preset data structure based on a preset template. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template.

[0163] The compression module 1420 is used to compress the log block into different storage units based on the preset data structure;

[0164] The retrieval module 1430 is used to locate the storage unit corresponding to the retrieval keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and to complete the retrieval of log entries in each log block based on the retrieval keyword based on the matching result between the retrieval keyword and the located storage unit.

[0165] The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into a local template based on the variable values ​​extracted from the variable intervals corresponding to the local template pattern.

[0166] The log processing system provided in this invention records the location information of sub-variables that need to be compressed in a log block as key points and metadata corresponding to the encoded stream in a preset data structure using a preset template. This allows the log blocks to be compressed into different storage units based on this preset data structure during log encoding and compression. By using the matching result of the search keyword and the decompressed preset template as a location vector, the system locates the storage unit corresponding to the search keyword based on the distribution of the location vectors of each log block on the location matrix. Based on the matching result of the search keyword and the located storage unit, the system completes the retrieval of log entries in each log block based on the search keyword. Low-latency retrieval at the storage unit level is achieved using vectorization technology. Furthermore, the preset template includes static and dynamic templates, with the dynamic template further including global and local template styles. This allows key points for compression to be marked during log block writing, and the local template style to be determined and filled with details based on the actual shape of the log block during actual compression. This not only minimizes copying during log writing but also ensures a high compression ratio.

[0167] Figure 15 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 15 As shown, the electronic device may include: a processor 1510, a communication interface 1520, a memory 1530, and a communication bus 1540. The processor 1510, communication interface 1520, and memory 1530 communicate with each other via the communication bus 1540. The processor 1510 can call logical instructions in the memory 1530 to execute a log processing method. This method includes: based on a preset template, recording the position information of sub-variables that need to be compressed in the log block as key points and metadata corresponding to the encoded stream in a preset data structure. The log block is obtained by segmenting the log. The preset template includes a static template and a dynamic template. Key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream consists of sub-variables appearing at the same position in the preset template. Based on the preset data structure, the log block is compressed into different storage units. Based on the distribution of the position vectors corresponding to each log block on the position matrix, the storage unit corresponding to the search keyword is located, and based on... The system retrieves the matching results between search keywords and located storage units to complete the retrieval of log entries in each log block based on the search keywords. The position vector is the matching result between the search keywords and the decompressed preset template. The static template is obtained by removing variable groups from the log samples. The dynamic template is trained based on variable groups. The dynamic template includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into the local template based on the variable values ​​extracted from the variable interval corresponding to the local template pattern.

[0168] Furthermore, the logical instructions in the aforementioned memory 1530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log processing method provided by the above methods. The method includes: recording the position information of the sub-variables that need to be compressed in the log block as key points and metadata corresponding to the encoded stream in a preset data structure based on a preset template; the log block is obtained by splitting the log; the preset template includes a static template and a dynamic template; the key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable; the encoded stream is composed of sub-variables that appear in the same position in the preset template; compressing the log block to different storage units based on the preset data structure; and compressing the log block based on the position vector corresponding to each log block. Based on the distribution of the location matrix, the storage unit corresponding to the search keyword is located, and the retrieval of log entries in each log block is completed based on the matching result of the search keyword and the located storage unit. The location vector is the matching result of the search keyword and the decompressed preset template. Among them, the static template is obtained by removing the variable group in the log sample; the dynamic template is trained based on the variable group. The dynamic template includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into the local template based on the variable values ​​extracted from the variable interval corresponding to the local template pattern.

[0170] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the log processing method provided by the above methods. This method includes: based on a preset template, recording the position information of sub-variables in a log block that need to be compressed as key points and metadata corresponding to the encoded stream into a preset data structure; the log block is obtained by segmenting the log; the preset template includes a static template and a dynamic template; the key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable; the encoded stream consists of sub-variables appearing at the same position in the preset template; based on the preset data structure, compressing the log block into different storage units; and based on the distribution of the position vectors corresponding to each log block on the position matrix, locating the corresponding... The system retrieves the storage unit corresponding to the search keyword and, based on the matching result of the search keyword and the located storage unit, completes the retrieval of log entries in each log block based on the search keyword. The position vector is the matching result of the search keyword and the decompressed preset template. The static template is obtained by removing the variable group from the log sample. The dynamic template is trained based on the variable group. The dynamic template includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into the local template based on the variable values ​​extracted from the variable interval corresponding to the local template pattern.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. < / v0>

Claims

1. A log processing method characterized by, include: Based on a preset template, the position information of the sub-variables that need to participate in compression in the log block is recorded as key points and metadata corresponding to the encoded stream in a preset data structure. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template. Based on the preset data structure, the log blocks are compressed into different storage units, specifically including: extracting the position information of sub-variables corresponding to the global template and local template styles based on the preset data structure; storing sub-variables of the same type consecutively in one storage unit based on the position information of sub-variables corresponding to the global template; and filling the sub-variables to a preset length and storing them consecutively in one storage unit based on the position information of sub-variables corresponding to the same local template. Based on the distribution of the location vectors corresponding to each log block on the location matrix, the storage unit corresponding to the search keyword is located, and based on the matching result between the search keyword and the located storage unit, the retrieval of log entries in each log block is completed based on the search keyword. The location vector is the matching result between the search keyword and the decompressed preset template. The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and the dynamic template includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into a local template based on the variable values ​​extracted from the variable intervals corresponding to the local template pattern. The global template is the template corresponding to the template with the largest number of templates among several template styles obtained by training the variable group; the local template styles are all other template styles except the template style corresponding to the global template.

2. The log processing method according to claim 1, characterized in that, The constant information and the type and length information are obtained based on the pattern of the global template and the variable group during training.

3. The log processing method according to claim 1, characterized in that, The preset data structure is a data structure that utilizes the prefetch function of the CPU cache for acceleration. The step of recording the location information of the sub-variables in the log block that need to participate in compression as key positions and corresponding metadata of the encoded stream into a preset data structure based on a preset template includes: The offset and length of each key point are stored in an entry of a preset interleaving table. Each row of the preset interleaving table is aligned with the CPU cache row of the preset data structure. Each row records multiple key points belonging to the same encoding stream and a pointer to the next row belonging to the same encoding stream. A contiguous array of metadata stores metadata corresponding to different encoded streams belonging to the same static template.

4. The log processing method according to claim 1, characterized in that, The step of locating the storage unit corresponding to the search keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and completing the retrieval of log entries in each log block based on the search keyword based on the matching result of the search keyword and the located storage unit, includes: Obtain search keywords; Based on the search keywords, match the decompressed preset template; The matching results are stored as position vectors in the position matrix; Based on the distribution of the location vectors corresponding to each log block on the location matrix, the location vectors are merged and / or pruned to locate the storage unit corresponding to the search keyword; Based on the search keywords, the storage units located are matched to obtain intermediate search results corresponding to each log block; The intermediate search results are merged and / or pruned to obtain the final search results, thus completing the search for log entries in each log block based on the search keywords.

5. The log processing method according to claim 4, characterized in that, The process of matching and locating the storage unit based on the search keywords to obtain intermediate search results corresponding to each log block includes: Based on the key parameters and the search keywords, unnecessary storage units are filtered out. The unnecessary storage units are those that do not contain the search keywords. The remaining storage units after filtering are decompressed, and when searching using an integer matching algorithm, the potential occurrence area of ​​the target content is determined based on the key parameters and the search keywords. The target content is the log entry stored in the storage unit that corresponds to the search keywords. Based on the search keywords, the target content is searched in the potential occurrence area to obtain the intermediate search results; The key parameters were determined using the following method: Data feature analysis is performed on the variables of the log extracted by the preset template to determine the data features of the variable data, including distribution features and changing trends; The data features are profiled to generate key parameters that reflect the characteristics of the variables.

6. The log processing method according to claim 5, characterized in that, The step of matching and locating the storage unit based on the search keywords to obtain intermediate search results corresponding to each log block further includes: Determine whether the remaining storage units after filtering are of dictionary type; After decompressing the dictionary index unit of the storage unit of the dictionary type, the text matching method is used. After a successful match, the dictionary value of the storage unit of the dictionary type is decompressed, and the integer matching algorithm is used for retrieval. For storage units that are not of dictionary type, the data type of the storage unit is determined. If the data type is text, a string matching algorithm is used for retrieval. If the data type is integer, an integer matching algorithm is used for retrieval.

7. The log processing method according to claim 4, characterized in that, The position matrix includes a first-level position matrix and a second-level position matrix; The first-level position matrix is ​​used to store the first position vector, which is the matching result of the decompressed static template based on the search keyword; the first position vector includes a variable identifier marking a successful match, the split search keyword, the matching alignment of the search keyword on the static template, and the matching status; The second-level position matrix is ​​used to store the second position vector, which is the matching result of the decompressed dynamic template based on the search keyword; the second position vector includes the variable identifier of the successful match, the split search keyword, and the matching alignment of the search keyword on the dynamic template; The static template is first decompressed and matched. If the matching status in the first position vector is successful, the search result is returned directly. Otherwise, the dynamic template is decompressed and matched.

8. The log processing method according to claim 4, characterized in that, The process of merging and / or pruning the intermediate search results to obtain the final search results includes: The intermediate retrieval results are merged and / or pruned based on a vectorized bitmap structure. The merged and / or pruned intermediate retrieval results are stored in the repository of the vectorized bitmap structure; Based on the data in the warehouse, the logs are restructured to obtain the final search results.

9. A log processing system, characterized in that, include: The writing module is used to record the position information of the sub-variables that need to be compressed in the log block as key points and metadata corresponding to the encoded stream into a preset data structure based on a preset template. The log block is obtained by splitting the log. The preset template includes a static template and a dynamic template. The key points are determined based on the dynamic template corresponding to the variable values ​​of each sub-variable. The encoded stream is composed of sub-variables that appear in the same position in the preset template. A compression module is used to compress the log block into different storage units based on the preset data structure; specifically, the compression module is used to: extract the position information of sub-variables corresponding to the global template and local template styles based on the preset data structure; and store the sub-variables of the same type consecutively in one storage unit based on the position information of the sub-variables corresponding to the global template. Based on the position information of the sub-variables corresponding to the same local template, the sub-variables are filled with a preset length and stored continuously in one storage unit; The retrieval module is used to locate the storage unit corresponding to the retrieval keyword based on the distribution of the location vectors corresponding to each log block on the location matrix, and to complete the retrieval of log entries in each log block based on the retrieval keyword based on the matching result of the retrieval keyword and the located storage unit. The location vector is the matching result of the retrieval keyword and the decompressed preset template. The static template is obtained by removing variable groups from the log samples; the dynamic template is trained based on the variable groups, and the dynamic template includes a global template and at least one local template pattern. The global template includes constant information, as well as the type and length information of the sub-variables included in the global template. The local template pattern includes the number of sub-variables and the delimiter information between each sub-variable. When compressing the log block, the corresponding local template pattern is filled into a local template based on the variable values ​​extracted from the variable intervals corresponding to the local template pattern. The global template is the template corresponding to the template with the largest number of templates among several template styles obtained by training the variable group; the local template styles are all other template styles except the template style corresponding to the global template.

Images