A log template extraction method, system and reinforcement method based on fixed writing mode mining

Abstract

The application belongs to but is not limited to the technical field of log data, and discloses a log template extraction method and system based on fixed writing mode mining and a reinforcement method. The method comprises the following steps: performing structural entity replacement preprocessing on original log text, realizing timestamp standardization and special character standardization; performing normalized coding on the preprocessing result, and converting a character stream into a structural semantic coding sequence; performing horizontal fixed writing mode identification on a single log based on a greedy strategy, and extracting a high-frequency coding sequence; performing extended sequence mining in a streaming log based on a light-heavy separation strategy, and realizing cross-log vertical mode reinforcement; and fusing the horizontal and vertical modes and completing template generation and deduplication based on similarity. The method uniformly maps multiple source heterogeneous logs to a coding sequence space, does not need to preset rules, and has self-adaptive template extraction capability.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of log data technology, and particularly relates to a log template extraction method, system, and enhancement method based on fixed writing pattern mining. Background Technology

[0002] Log data is the core recorder of the operational status of modern computer systems and network devices, widely generated in various hardware and software components such as operating systems, middleware, databases, and security auditing systems. With the widespread adoption of microservice architecture and cloud-native technologies, enterprise-level systems generate billions of log entries daily, originating from diverse sources and formats, forming a typical multi-source heterogeneous characteristic. Log template extraction, which automatically identifies constant skeleton structures from unstructured or semi-structured raw log text and generates reusable parameterized templates, is a fundamental capability of log management platforms, security information and event management systems, and intelligent operation and maintenance platforms. It plays an irreplaceable supporting role in application scenarios such as fault diagnosis, security auditing, and compliance testing.

[0003] Early log parsing methods based on manual rules relied on domain experts pre-writing regular expressions or syntax rules for specific log formats, offering advantages such as high parsing accuracy and strong controllability. However, the cost of writing and maintaining manual rules is extremely high. Faced with the frequent evolution of log formats and the continuous incorporation of new log sources, manual updates to the rule base are insufficient to meet practical engineering needs, resulting in significant bottlenecks in scalability. Log parsing methods, represented by automated clustering and sequence alignment, automatically discover log templates without relying on prior rules by designing specific clustering rules or sequence alignment strategies. However, these methods generally suffer from limitations such as variable identification strategies being mostly based on character-level lexical features and limited generalization ability. They also lack a dedicated structure-aware mechanism when dealing with the common fixed writing patterns in logs, leading to insufficient variable extraction accuracy. Recently emerging log parsing methods based on pre-trained language models have improved parsing generalization by leveraging the semantic understanding capabilities of large models. However, these methods have extremely high requirements for computing resources and large inference latency, making it difficult to meet the engineering deployment needs of large-scale real-time log processing. In addition, the black-box nature of the model leads to insufficient interpretability of the parsing results, which poses obstacles to its use in scenarios with strict credibility requirements, such as security auditing.

[0004] Existing log template extraction methods mainly suffer from the following problems: (1) Methods based on manual rules rely on manually writing rules, which has high maintenance costs and poor scalability; (2) The clustering and statistical methods have insufficient accuracy in variable identification, weak ability to perceive the structure of fixed writing patterns in logs, and limited generalization ability; (3) Deep learning-based methods have high computational overhead and poor interpretability, making it difficult to meet the engineering deployment requirements of large-scale real-time log processing.

[0005] Therefore, how to propose a solution to the above problems is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a log template extraction method and system based on fixed writing pattern mining, as well as a vertical fixed writing pattern enhancement method for streaming logs.

[0007] This invention is implemented as follows: a log template extraction method based on fixed writing pattern mining, comprising: The original log text is processed by replacing fixed-structure entities to obtain normalized log text containing structure entity placeholders; The normalized log text is subjected to character-level structural semantic encoding to obtain an encoding sequence composed of structural semantic tags and content placeholders; The encoding sequence of a single log entry is subjected to horizontal fixed writing pattern recognition to obtain the horizontal fixed writing pattern that appears repeatedly within the log. The horizontal fixed writing pattern is used as a compression unit to perform horizontal compression on the encoded sequence in the streaming log, and cross-log candidate pattern mining is performed on the compressed sequence to obtain the vertical fixed writing pattern. Based on the horizontal fixed writing pattern and the vertical fixed writing pattern, the boundaries of structural fields are located in the log encoding sequence, a parameterized log template is generated, and the parameterized log template is deduplicated by similarity aggregation.

[0008] Furthermore, the fixed writing structure entity replacement process includes: compressing the original log text with whitespace characters; matching multiple timestamp formats according to a preset priority; replacing the matched timestamps with numbered timestamp placeholders; and establishing a mapping relationship between the timestamp placeholders and the original timestamp strings.

[0009] Furthermore, the character-level structural semantic encoding includes: retaining timestamp placeholders as indivisible atomic units; replacing special characters with structural semantics with corresponding structural semantic tags; merging consecutive ordinary text character segments into content placeholders; and merging consecutive whitespace tags into whitespace structural tags of different granularities according to their quantity.

[0010] Furthermore, the horizontal fixed writing pattern recognition includes: counting the frequency of consecutive tag pairs in a single log encoding sequence using a preset length window; removing tag pairs whose starting tags belong to the invalid starting tag set; determining the order of candidate seeds according to frequency and first occurrence position; and performing greedy look-ahead extension on the candidate seeds to obtain candidate horizontal fixed writing patterns.

[0011] Furthermore, the greedy look-ahead extension includes: recording all starting positions of the candidate seed in the encoded sequence; counting the frequency of occurrence of the successor label at the current position; selecting the successor label that meets the frequency condition and is optimally sorted and appending it to the candidate seed; updating the set of starting positions that can still match the extended sequence; and stopping the extension when there is no successor label that meets the condition or when the maximum length is reached.

[0012] Furthermore, the generation of the parameterized log template includes: using the relational label or structure separator label as the pivot point, performing field boundary expansion on both sides of the candidate pattern matching position; resolving bit conflicts for the identified key-value pair candidates; performing supplementary pivot scanning on the log interval not covered by the candidate pattern; performing typed replacement on implicit variables; and generating the parameterized log template according to the final replacement range.

[0013] Furthermore, the implicit variables include Uniform Resource Locators (URLs), file paths, network address and port combinations, network addresses, unique identifiers, and plain numeric strings; the implicit variables are identified and replaced with numbered typed placeholders according to a preset priority.

[0014] Furthermore, the similarity aggregation deduplication includes: calculating the edit distance or encoding sequence overlap rate between parameterized log templates; classifying parameterized log templates with similarity reaching a preset threshold into the same template cluster; selecting the parameterized log template with the largest coverage in the template cluster as the representative template; and establishing a hierarchical index between the representative template and the corresponding template cluster.

[0015] This invention also provides a method for enhancing the vertical fixed writing pattern of streaming logs, comprising: Obtain the log encoding sequence and its corresponding horizontal fixed writing pattern; After sorting the horizontal fixed writing patterns by length, perform exact matching in the log encoding sequence, and replace the matched tag segments with pattern placeholders to obtain the compressed encoding sequence; A sliding window enumeration is performed on the compressed encoding sequence, and the pattern placeholders in the window are expanded into the corresponding original encoding sequences to obtain candidate vertical patterns. The candidate longitudinal patterns are validated, and the validated candidate longitudinal patterns are written into the light and heavy separation frequency statistics structure. Based on the frequency results in the frequency statistics structure for separating light and heavy elements, output the vertical fixed writing pattern that meets the support condition.

[0016] Furthermore, the light-heavy separation frequency statistics structure includes a heavy mass bucket and a light mass bucket; the heavy mass bucket is used to store the precise count and defense value of candidate longitudinal patterns; the light mass bucket is used to store the approximate frequency of candidate longitudinal patterns that have not entered the heavy mass bucket; when the target bucket of the heavy mass bucket is full, it is determined whether to replace the existing patterns in the target bucket based on the defense value, approximate frequency and precise count, and the defense value in the heavy mass bucket is attenuated in batches.

[0017] This invention also provides a log template extraction system based on fixed writing pattern mining, comprising: The structure entity replacement module is used to perform fixed-writing structure entity replacement processing on the original log text to obtain normalized log text containing structure entity placeholders. The character-level structural semantic encoding module is used to perform character-level structural semantic encoding on the normalized log text to obtain an encoding sequence composed of structural semantic tags and content placeholders; The horizontal fixed writing pattern recognition module is used to recognize the horizontal fixed writing pattern of the encoded sequence of a single log and obtain the horizontal fixed writing pattern that appears repeatedly within the log. The vertical fixed writing pattern mining module is used to use the horizontal fixed writing pattern as a compression unit to perform horizontal compression on the encoded sequence in the streaming log, and to perform cross-log candidate pattern mining on the compressed sequence to obtain the vertical fixed writing pattern. The log template generation module is used to locate the structural field boundaries in the log encoding sequence based on the horizontal fixed writing pattern and the vertical fixed writing pattern, generate a parameterized log template, and perform similarity aggregation to remove duplicates from the parameterized log template.

[0018] Furthermore, the structural entity replacement module is used to compress whitespace characters in the original log text, match multiple timestamp formats according to a preset priority, replace the matched timestamps with numbered timestamp placeholders, and establish a mapping relationship between the timestamp placeholders and the original timestamp strings; the character-level structural semantic encoding module is used to retain timestamp placeholders as indivisible atomic units, replace special characters with structural semantics with corresponding structural semantic tags, merge continuous ordinary text character segments into content placeholders, and merge continuous whitespace tags into whitespace structural tags of different granularities according to their quantity; the horizontal fixed writing pattern recognition module is used to count the frequency of continuous tag pairs in a single log encoding sequence with a preset length window, remove tag pairs whose starting tags belong to the invalid starting tag set, determine the candidate seed order according to frequency and first occurrence position, and perform greedy look-ahead extension on the candidate seeds to obtain candidate horizontal fixed writing patterns.

[0019] In view of the technical problems existing in the prior art and the difficulty of solving these problems, this paper closely combines the technical solution to be protected by this invention with the results and data from the research and development process, and analyzes in detail how the technical solution of this invention solves the technical problems, as well as the inventive technical effects brought about by solving these problems. The specific description is as follows: To address the common problems of existing log template extraction methods when dealing with multi-source heterogeneous logs, such as significant format differences, inconsistent character representations, and difficulty in directly comparing stable structures, this invention first performs timestamp placeholder replacement, special character classification, and structural semantic tagging encoding on the original log text. This maps logs from different sources and with different formats into a unified structural semantic encoding sequence. Because this processing method preserves log structural information while mitigating differences in specific character values, it effectively reduces the interference of character-level noise on template extraction. This results in logs with the same writing structure but different text content exhibiting higher consistency under a unified tag space, thus providing a stable foundation for subsequent fixed writing pattern recognition and template generation. Therefore, this invention achieves adaptive template extraction from multi-source heterogeneous logs without requiring pre-written manual rules for specific log sources, improving the consistency and stability of template generation under unsupervised conditions.

[0020] To address the problem that existing technologies, relying solely on single clustering, statistical, or sequence alignment strategies, struggle to simultaneously identify both the ability to recognize repetitive local structures within a single log entry and the ability to utilize global common patterns across logs, this invention collaboratively mines fixed writing patterns from both horizontal and vertical dimensions. On the one hand, a position-aware greedy extension algorithm is employed horizontally to screen candidate seeds, perform greedy look-ahead extension, and verify structural validity for the encoded sequence of a single log entry, identifying recurring horizontal fixed writing patterns within the log. On the other hand, by introducing horizontal patterns as compression units into the cross-log candidate pattern mining process, the ability to identify stable common structures across logs is further enhanced. Since horizontal patterns can capture frequently repetitive local writing skeletons within a single log entry, and vertical patterns can supplement and strengthen more stable global common structures across logs, their synergistic effect effectively improves the completeness of template extraction, field boundary identification ability, and parsing stability in heterogeneous log environments. Compared to existing methods that rely solely on a single statistical path, this invention can more fully utilize structural information within and between log entries, exhibiting better template generalization ability and structural recognition accuracy. To address the challenges of high storage overhead and limited retention of high-frequency patterns in traditional precise statistical methods due to the continuously growing log size and large number of candidate patterns in streaming log processing scenarios, this invention introduces a frequency statistical structure based on a heavy-weight separation strategy. This structure continuously tracks high-frequency patterns and achieves dynamic competition and replacement of candidate patterns through a defense force value and approximate frequency coordination mechanism. Since the heavy-weight bucket can accurately count important high-frequency patterns, the light-weight bucket can approximate the estimation of low-frequency candidate patterns, and the defense force decay mechanism can gradually remove long-term undetected low-frequency historical patterns, this invention can still stably identify and retain true high-frequency patterns even under limited storage resources. Therefore, this invention not only effectively utilizes cross-log frequency information to strengthen and supplement the lateral identification results but also reduces computational overhead and storage pressure in streaming processing scenarios, improving the real-time processing capability and engineering deployment feasibility of the method. This balance between high-frequency pattern retention and online processing capability under resource constraints is not something that existing simple statistical methods can naturally achieve. To address the common technical problems in existing log template extraction technologies, such as high template redundancy, difficulty in uniformly merging similar logs, unclear boundaries between explicit fields and implicit variables, and insufficient interpretability of parsing results, these problems are even more difficult to handle in multi-source heterogeneous log scenarios due to large differences in log sample expression, strong local text fluctuations, and multiple expressions of the same semantic content. This invention, based on the horizontal fixed writing pattern recognition and vertical fixed writing pattern enhancement, further adopts a template fusion and deduplication technology based on similarity measurement. It performs structural similarity analysis, hierarchical aggregation, and representative template selection on the generated parameterized template set, thereby uniformly merging templates with similar structures but different surface expressions, reducing template redundancy and improving the standardization of template library organization. Meanwhile, during template fusion, this invention combines field boundary expansion driven by relational or structural delimiters, occupancy conflict resolution, and implicit variable post-processing identification mechanisms to collaboratively extract explicit and implicit fields from logs. Specifically, for fields in templates that are not yet explicitly parameterized but exhibit stable positional distributions with varying values ​​at the same location in similar templates, this invention further identifies these implicit variables and performs parameterized replacement by statistically analyzing the term frequency distribution at corresponding positions in aggregated similar templates. Through the above technical solutions, this invention not only effectively eliminates redundant templates caused by differences in log samples, improving the structural consistency and field partitioning accuracy after template fusion, but also enhances the ability to identify implicit structural fields. This allows the output results to be presented in the form of a standard key-value pair dictionary and parameterized templates, possessing high readability and interpretability, facilitating manual verification and integration with downstream business systems. Thus, it achieves a comprehensive technical effect that balances template simplification, structured field expression, and interpretable output.

[0021] (1) The technical solution of the present invention solves a technical problem that people have long wanted to solve but have never been able to solve successfully: The technical solution of this invention solves a long-standing technical problem in the field of log template extraction: how to achieve template extraction with strong structure awareness, field boundary identification, and efficient streaming processing for multi-source heterogeneous logs without relying on a large number of manual rules and highly complex models. Existing technologies typically struggle to simultaneously achieve parsing accuracy, generalization ability, real-time performance, and interpretability. This invention, however, effectively solves these technical challenges through collaborative techniques such as structural semantic tag encoding, horizontal fixed writing pattern recognition, vertical common pattern enhancement, and template fusion deduplication.

[0022] (2) The technical solution of the present invention overcomes technical bias: The technical solution of this invention overcomes the technical bias in the field that "high-precision log parsing must rely on manual rules or highly complex semantic models." By standardizing and encoding log structural features, and jointly utilizing the repetitive structure within a single log and the common structure across logs for bidirectional pattern mining, this invention can still achieve effective template extraction and structured parsing of heterogeneous logs without the need for a pre-set rule base or large-scale training models. This demonstrates that the technical approach based on structure pattern awareness can also achieve good technical results. Attached Figure Description

[0023] Figure 1 This is a flowchart of the log template extraction method based on fixed writing pattern mining in this embodiment of the invention.

[0024] Figure 2 This is a schematic diagram of the standardized encoding hierarchical processing flow for the diverse constituent elements of log text in an embodiment of the present invention.

[0025] Figure 3 This is a flowchart of a single log internal fixed writing pattern recognition algorithm based on a greedy strategy in an embodiment of the present invention.

[0026] Figure 4 This is a flowchart of the streaming log extended coding sequence mining method based on the light and heavy separation strategy in this embodiment of the invention.

[0027] Figure 5 This is a performance comparison chart of the embodiments of the present invention and various log parsers under the FGA metric.

[0028] Figure 6 This is a performance comparison chart of the embodiments of the present invention and various log parsers under the FTA metric.

[0029] Figure 7 This is a comparison chart of the comprehensive ranking of the embodiments of the present invention and various log parsers under multi-dimensional evaluation indicators.

[0030] Figure 8 This is a comparison chart of the parsing accuracy of embodiments of the present invention and various log parsers on a partial dataset.

[0031] Figure 9 This is a schematic diagram illustrating the effect of lateral mode compression on sequence length and vertical insertion operation provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] The log template extraction method based on fixed writing pattern mining provided in this embodiment of the invention includes: S1: Perform preprocessing on the original log text based on fixed writing structure entity replacement, and complete timestamp standardization and special character normalization; S2: Perform normalized encoding on the preprocessed log text for diverse components, transforming the character stream into a structured semantic encoding sequence; S3: For the encoding sequence of a single log entry, a greedy strategy is used to perform fixed writing pattern recognition within the log entry and extract high-frequency encoding sequences horizontally; S4: Based on the light and heavy separation strategy, extended coding sequence mining is performed in streaming logs to achieve vertical common pattern reinforcement across logs; S5: Integrate the results of horizontal and vertical pattern mining, and perform template fusion and deduplication based on similarity metrics.

[0034] Furthermore, S1 specifically includes: S1.1: Timestamp standardization and variable placeholder replacement, specifically including the following sub-steps: S1.1.1: Text format standardization; Perform whitespace character compression on the original log text, and use regular expressions to uniformly replace multiple consecutive whitespace characters with a single space to eliminate the problem of inconsistent whitespace format caused by differences in log sources; S1.1.2: Multi-mode timestamp recognition; according to a preset priority order, each rule in the timestamp regular expression pattern library is applied to the log text one by one. The pattern library covers more than ten mainstream formats, including ISO 8601 format, standard date and time format and its millisecond variant, month abbreviation format, Unix timestamp with milliseconds and sequence number extension, pure time format, dotted date format and compact special format; each rule is configured with negative look-ahead assertion and prefix negative look-back assertion to prevent file paths, version numbers and other numeric strings from being mismatched; S1.1.3: Placeholder generation and mapping records; For each successfully matched timestamp, the original value is encoded in UTF-8 and converted into a hexadecimal string, which is then embedded in the text as an internal temporary marker; After all pattern matching is completed, unified post-processing is performed to replace all temporary markers with a format like...<TIMESTAMP_n> The numbered placeholders are recorded one by one in the external mapping table, and the correspondence between the placeholder key and the original timestamp string value is recorded for accurate querying when restoring the subsequent parsing results; S1.2: Special character classification and annotation, specifically including the following sub-steps: S1.2.1: Definition of special character set; a predefined structural special character set, which covers more than twenty characters with structural semantics in log text, including bracket characters, relational operators, delimiters, modifiers, quotation mark characters, whitespace characters, and slash characters; S1.2.2: Character classification; Perform character-level scanning on the log text after the timestamp placeholders are replaced, and classify each character in the text into one of the three categories: timestamp placeholders, special characters to be encoded, and ordinary text characters, so as to provide deterministic input for the standardized encoding in step S2.

[0035] Furthermore, step S2 specifically includes: S2.1: Lexical unit abstraction encoding, specifically including the following sub-steps: S2.1.1: Timestamp placeholder atomic preservation; Scan the log text character by character, and when the current scan position can match the complete string of the numbered timestamp placeholder, append the placeholder as an indivisible atomic unit to the output encoding sequence and skip the character position of the corresponding length; S2.1.2: Structural Special Character Replacement; For all other characters except the timestamp placeholder, query the predefined structural special character mapping table one by one; if the current character exists in the mapping table, replace it with the corresponding named semantic tag, including... <lparen> 、 <rparen> 、 <equals> 、 <colon> 、 <comma> 、 <space> 、 <quote> 、 <slash>More than twenty types of tags; if the current character is not in the mapping table, it will be preserved as a plain text character; S2.1.3: Merging consecutive plain text; Perform a second traversal on the intermediate encoded sequence after processing S2.1.1 to S2.1.2, and merge consecutive plain text character segments into a single content placeholder. <content>This transforms the character stream into a stream composed of structural semantic tags and <content>Alternating encoded sequences; S2.2: Multi-granularity semantic feature merging, specifically including the following sub-steps: S2.2.1: Counting consecutive blank tags; traverse the encoded sequence output from S2.1 again to identify consecutive blank tags. <space>Tag segment: Count the number of consecutive blank tags in each segment; S2.2.2: Multi-granularity whitespace normalization; performs merging based on the number of consecutive whitespace tags: four or more consecutive whitespace tags. <space>Tag merging and replacement with single <tab>Tags; two consecutive occurrences <space>Tag merging and replacement with single <doublespace>Labels; the remaining number of consecutive blank labels are retained unchanged.

[0036] S3 specifically includes: S3.1: Horizontal high-frequency sequence seed discovery, specifically including the following sub-steps: S3.1.1: Binary Tag Pair Frequency Statistics; Traverse the encoded sequence of the current log using a sliding window of length 2, count the frequency of each consecutive tag pair, and record the index of the first occurrence of each tag pair in the sequence, where the starting tag is... <space>Window skipping is not counted; S3.1.2: Candidate Seed Screening and Sorting; Valid candidate seeds are screened according to two rules: First, the frequency of occurrence of the tag pair is not lower than a preset minimum frequency threshold; Second, the starting tag of the tag pair does not belong to the invalid starting tag set, which includes... <space> 、 <equals> 、 <rparen> 、 <rbrace> 、 <rbracket> 、 <dot> 、 <colon>Labels that are not suitable as the starting point of a pattern are removed; the selected candidate seeds are arranged in descending order of frequency and ascending order of first occurrence position to form an ordered candidate queue. S3.2: Seed expansion based on greedy look-ahead algorithm, specifically including the following sub-steps: S3.2.1: Seed starting position location; For each seed in the ordered candidate queue, scan the complete encoded sequence and record the starting position index list of all occurrences of the seed in the sequence, as the location basis for subsequent look-ahead expansion; S3.2.2: Greedy look-ahead extension; Enter extension loop: Based on the current starting position list, count the frequency of the next label immediately following the end of the sequence at each position; if there are candidate extension labels with a frequency not lower than the minimum threshold, select the label with the highest frequency and the earliest first appearance position as the optimal extension direction, append it to the end of the current sequence, and filter the starting position list to a subset of positions that can still match the extended new sequence; repeat this loop until there are no extension candidates that meet the conditions or the length of the current sequence reaches the preset maximum value; the original label sequence is not modified during the scanning process; S3.2.3: Validation and Output of Extended Results; The following validations are performed sequentially on the extended sequence: First, the actual frequency of occurrence of the sequence in the original encoded sequence is recounted in an overlapping manner to verify whether it still meets the minimum frequency requirement; then, invalid end tags at the end of the sequence are cyclically trimmed, wherein the set of invalid end tags contains <space> 、 <doublespace> 、 <underscore> 、 <hyphen> 、 <dot>Then perform structural validity checks, including checks on bracket pairing, quotation mark pairing, content existence, and component uniqueness; finally, verify that the length of the trimmed sequence is greater than 2; output the sequence that passes all checks as a valid horizontal fixed writing mode and record its frequency.

[0037] S4 specifically includes: S4.1: Frequency statistics structure initialization and lateral compression based on the light-heavy separation strategy, specifically including the following sub-steps: S4.1.1: Initialization of the heavy-weight separation structure; Initialize the probability approximate frequency statistical structure based on the heavy-weight separation strategy. This structure consists of two parts: a heavy-weight bucket and a light-weight bucket. The heavy-weight bucket is a hash bucket array, with each bucket containing a fixed number of cells. Each cell stores the exact occurrence count and defense value of an encoded sequence. The defense value is a count value used to measure the stability of the pattern occupying the bucket position. The light-weight bucket is a minimum frequency matrix, consisting of multiple rows of hash tables, used to approximate the frequency of low-frequency candidate patterns that cannot enter the heavy-weight bucket. S4.1.2: Horizontal pattern placeholder compression; For each log to be processed, the horizontal fixed writing pattern output in step S3 is arranged in descending order of length, and precise matching is performed in the encoding sequence of the log in turn. The pattern label segment at each matching position is replaced with a placeholder with a unique number, and a mapping table from placeholder to original pattern label tuple is maintained, thereby compressing the encoding sequence into a short sequence containing placeholders. S4.2: Sliding window candidate extraction and insertion of frequency statistics for light and heavy separation, specifically including the following sub-steps: S4.2.1: Compressed sequence sliding window enumeration; Using a preset minimum to maximum length as the window range, perform a full-size sliding window operation on the compressed sequence, enumerating all candidate subsequence windows one by one; For each candidate window, expand the placeholders in it to restore the corresponding original label tuples, and obtain the fully expanded candidate pattern; S4.2.2: Candidate pattern validity check; Perform validity check on the expanded candidate patterns, requiring them to contain either a relational label or a structural delimiter label, and not to contain a predefined set of excluded subsequences; Patterns that pass the check are then inserted into the light-heavy separation frequency statistics structure insertion process; S4.2.3: Light and Heavy Separation Structure Insertion Decision; For each candidate pattern that passes the verification, perform an insertion operation: Hash the candidate pattern to the target bucket in the heavy mass bucket array. If a cell with the same pattern already exists in the bucket, increment the cell count and reset the defense value to the initial value. If an empty cell exists in the bucket, occupy the empty cell and initialize the count and defense value. If the bucket is full, select the cell with the smallest defense value as the candidate replacement target: When its defense value is greater than zero, decrement the value by one as a penalty, and add the new pattern to the minimum frequency matrix of the light mass bucket. When its defense value is zero, add the new pattern to the minimum frequency matrix and estimate its frequency. If the estimated frequency is higher than the exact count of the candidate cell, replace the cell with the new pattern and increment the estimated frequency by one as the initial exact count; otherwise, maintain the status quo. S4.3: Incremental output and defense attenuation, specifically including the following sub-steps: S4.3.1: Incremental output of high-frequency mode; After completing the statistical modeling of the frequency separation of light and heavy logs in the current batch, scan all cells of the heavy quality bucket, extract the patterns whose frequency of occurrence reaches the minimum support threshold, sort them in descending order of frequency and descending order of sequence length, and output them as the vertical fixed writing pattern set of the current batch; The processing flow supports streaming input, and the output can be triggered after each batch is processed, without waiting for the full log to arrive; S4.3.2: Defense force periodic decay; Perform a decrement operation on the defense force value of all non-empty cells in the heavy mass bucket according to the batch periodicity, so that the defense force of low-frequency historical patterns that have not been hit for a long time is gradually reduced to zero, thereby providing space for the promotion of high-frequency new patterns and ensuring the tracking accuracy of the current high-frequency patterns in the continuous streaming operation of the data structure.

[0038] S5 specifically includes: S5.1: Template structure similarity calculation, specifically including the following sub-steps: S5.1.1: Candidate pattern integration and sorting; horizontal patterns are sorted in descending order of the number of key symbol tags they contain, and vertical patterns are sorted in descending order of global frequency. The two are merged into a candidate pattern list to ensure that patterns with high complexity and high frequency participate in key-value pair extraction first. S5.1.2: Locating the position of the pattern in the log; Traversing the encoding sequence of the current log in a sliding window manner, locating the specific start and end positions of each candidate pattern one by one, providing a precise positional basis for subsequent pivot point expansion; S5.1.3: Pivot-point driven key-value pair expansion; for each candidate pattern's matching position in the log, for example, using <equals>or <colon>Using the labels as the pivot point, the recognition key field is expanded to the left, and the recognition value field is expanded to the right. During the expansion process, adjacent blank labels on both sides of the pivot are skipped. The expansion stops when a boundary character or an area already marked is encountered to the left. To the right, the recognition of values ​​enclosed in quotes and brackets is supported. <colon>The core component additionally detects and excludes IP address and port formats to prevent misidentification. S5.1.4: Occupancy bit conflict resolution; All identified key-value pair candidates are sorted by greedy priority based on the number of key symbol tags they contain, and an occupancy bit array is used for conflict detection to ensure that each tag position is covered by at most one key-value pair, eliminating field boundary ambiguity caused by multi-mode overlap; S5.2: Redundant template merging and simplification, specifically including the following sub-steps: S5.2.1: Global supplementary scan; After field extraction based on known patterns is completed, the axis-centric expansion retrieval is re-executed on the encoded sequence of the entire log to capture key-value pair structures not covered by any horizontal or vertical patterns, further improving the recall rate of field extraction; S5.2.2: Implicit variable post-processing identification; perform implicit variable identification on the initially generated template, and identify implicit variables that have not yet been parameterized in the template in the priority order of URL, file path, IP address and port pairing, single IPv4 / IPv6 address, UUID and pure number, replace them with numbered named type labels, and write the identified original values ​​into the parsing result dictionary. S5.3: Template hierarchical organization and indexing, specifically including the following sub-steps: S5.3.1: Parameterized template generation; Based on the final determined key-value pair replacement range, the corresponding tag range is replaced with parameterized placeholders, and the rest is restored to the original text, and the structured log template is generated by splicing them together. S5.3.2: Similarity-driven template aggregation and deduplication; The generated template set is hierarchically aggregated according to structural similarity. The similarity is measured by edit distance or coding sequence overlap rate. Templates with similarity exceeding a preset threshold are merged into the same cluster. The template with the widest coverage in the cluster is used as the representative template of the cluster, thereby eliminating redundant templates caused by differences in log samples and building a hierarchical index to support subsequent fast matching.

[0039] like Figure 1 As shown in the figure, this invention provides a log template extraction method based on fixed writing pattern mining. First, the timestamp field in the original log is subjected to multi-pattern recognition and placeholder replacement. Then, through lexical unit abstraction encoding and multi-granularity semantic feature merging, the log text is transformed into a structured semantic encoding sequence. On this basis, fixed writing patterns are mined from both horizontal and vertical dimensions. Then, using relational symbols or delimiters as the axis points, combined with the placeholder conflict resolution mechanism, unambiguous key-value pair fields are extracted, and the remaining implicit variables are post-processed for identification. Finally, through similarity-driven template fusion and deduplication, the structured key-value pair parsing results and generalized log templates are output.

[0040] like Figure 1 As shown, the log template extraction method based on fixed writing pattern mining proposed in this invention is implemented as follows: Step S1: Perform preprocessing on the original log text using entity replacement based on a fixed writing structure.

[0041] Step S1 aims to eliminate format noise in the log text caused by differences in source and extract highly variable fields such as timestamps into reversible structural placeholders, thereby providing a consistent input for subsequent encoding steps.

[0042] First, a format normalization operation is performed to compress redundant consecutive whitespace characters in the original log into a single space, eliminating inconsistencies in whitespace format caused by differences in log collection links. Then, a multi-pattern priority matching strategy is used to identify and replace timestamp fields in the logs. The timestamp regular expression pattern library covers more than ten mainstream formats, including the ISO 8601 standard format, standard date and time formats and their millisecond variants, month abbreviation formats, Unix timestamps with milliseconds and sequence number extensions. Disambiguation between patterns is achieved through negative look-ahead assertions and prefix negative look-back assertions to prevent mismatches of non-timestamp content containing numbers, such as file paths. Successfully identified original timestamp values ​​are temporarily stored in hexadecimal encoding in internal markers. In the post-processing stage, these are restored to numbered placeholders, and a one-to-one mapping relationship is established between the placeholders and the original values ​​to support accurate restoration of the original timestamp text in the parsing results. The output of step S1 is text with a standardized format and parameterized timestamps. Its direct function is to convert the most variable fields into a fixed form while preserving the log structure information, so that the encoding result of step S2 can maintain structural consistency among similar logs, thereby providing effective support for the horizontal pattern statistics in step S3.

[0043] Step S2: Perform normalization encoding on the preprocessed log text for its diverse components.

[0044] The purpose of step S2 is to transform the heterogeneous character streams into a structured sequence composed of discrete semantic tags, eliminate the interference of character-level differences on subsequent pattern statistics, and retain the position alignment information inside the log through multi-granularity blank merging.

[0045] like Figure 2 As shown, the normalized encoding is executed in two stages. The first stage is lexical unit abstraction encoding: each character in the log text is scanned one by one. If the current position matches a numbered timestamp placeholder, it is retained as an atomic unit. If the current character belongs to a predefined set of structural special characters (including bracket characters, relational characters, delimiters, modifiers, quotation marks, whitespace characters, and forward slashes, etc.), it is replaced with the corresponding named semantic tag. Otherwise, it is retained as a plain text character. The encoded sequence after the first stage is traversed a second time to merge consecutive plain text characters into a single content placeholder. <content>The second stage is multi-granularity feature merging: consecutive whitespace tags in the encoded sequence are counted and merged. Four or more consecutive whitespace tags are merged into tab tags, exactly two consecutive whitespace tags are merged into double-space tags, and the original number of whitespace tags is retained in other cases, giving the encoded sequence multi-granularity whitespace distribution features. The output of step S2 is a structured semantic encoded sequence, whose direct function is to project log texts from different sources onto a unified tag space, so that fragments with the same writing structure in different logs are mapped to the same tag subsequence. This enables the frequency statistics in step S3 to identify structural repetitions across specific character values, laying the foundation for the discovery of fixed writing patterns.

[0046] Step S3: For the encoded sequence of a single log entry, perform fixed writing pattern recognition within the log entry based on a greedy strategy.

[0047] The purpose of step S3 is to discover recurring structural fragments, i.e. horizontal fixed writing patterns, from the encoding sequence of a single log entry, thereby identifying the regular skeleton of the log in terms of writing form and providing structural priors for the accurate positioning of subsequent key-value pair fields.

[0048] like Figure 3 As shown, the horizontal fixed writing pattern recognition is performed in two stages. The first stage is seed discovery: the frequency of occurrence of all consecutive label pairs of length 2 in the encoded sequence is counted using a sliding window method, and the first occurrence position of each label pair is recorded; candidate seeds are selected based on two rules: the occurrence frequency is not lower than the preset minimum frequency threshold and the starting label does not belong to the invalid starting label set. The candidate seeds are arranged in descending order of frequency and ascending order of the first occurrence position to form an ordered candidate queue. The second step is greedy extension and verification: For each seed in the ordered candidate queue, first locate all its starting positions in the encoded sequence; then enter the extension loop, and in each iteration, count the frequency of the next tag immediately following each position based on the current starting position list. If there is a candidate extension tag that meets the minimum frequency requirement, select the tag with the highest frequency and the earliest first appearance and append it to the end of the current sequence while simultaneously filtering the starting position list. This iteration continues until it is impossible to extend further or the sequence length reaches the preset upper limit. During the scanning process, the original encoded sequence is not modified. After the extension is completed, the resulting sequence is subjected to overlap frequency verification, invalid tag pruning at the end, structural legality verification (including bracket pairing check, quotation mark pairing check, content existence check, and component singleness check), and length verification in sequence. The sequence that passes all verifications is output as a valid horizontal fixed writing pattern and its frequency is recorded. The output of step S3 is a set of high-frequency structural tag subsequences within the log. Its direct function is to provide a horizontal pattern for cross-log reinforcement to step S4, and to provide structural positioning clues for driving the expansion of the pivot point to step S5.

[0049] Step S4: Based on the light-heavy separation strategy, perform extended encoded sequence mining in the streaming log.

[0050] The purpose of step S4 is to enhance the lateral pattern obtained in step S3 by using the frequency accumulation information across logs, and to discover longer chain-like fixed writing patterns in a single log that cannot be identified individually due to insufficient sample size, thereby making up for the omission defect of lateral identification in logs with low repetition rate.

[0051] like Figure 4 As shown, the vertical cross-log common pattern mining is performed in three stages. The first stage is horizontal compression: for each log to be processed, the horizontal fixed writing patterns output in step S3 are arranged in descending order of length, and precise matching is performed on the encoding sequence of the log in turn. The pattern label segment at each matching position is replaced with a uniquely numbered placeholder, and a mapping table from placeholders to original pattern label tuples is maintained. The encoding sequence is compressed into a short sequence containing placeholders. Horizontal compression allows known horizontal patterns to participate in subsequent sliding window operations as atomic units, effectively expanding the visible length range of vertical candidate patterns. The second step is sliding window candidate extraction and insertion: using a preset minimum to maximum length as the window range, a full-size sliding window operation is performed on the compressed sequence, enumerating all candidate subsequence windows one by one. For each candidate window, the placeholders are expanded and restored to the corresponding original label tuples. The expanded candidate patterns are validated (requiring either a relational label or a structural separator label, and not containing a predefined excluded subsequence set). For patterns that pass the validation, a light-heavy separation structure insertion decision is made: the candidate pattern is hashed and mapped to the target bucket in the heavy mass bucket array. Based on the defense value, the decision is made to directly increment the count, occupy an empty cell, or cooperate with the light mass bucket to complete the competitive replacement, thereby accurately tracking high-frequency patterns in the sublinear space. The third step involves incremental output and defense attenuation: After processing each batch of logs, all cells in the heavy quality bucket are scanned, and patterns whose frequency reaches the minimum support threshold are extracted, sorted in descending order of frequency and length, and output as the vertical fixed writing pattern set for the current batch. Simultaneously, the defense value of all non-empty cells in the heavy quality bucket is decreased by one according to the batch cycle, gradually reducing the defense value of low-frequency historical patterns that have not been hit for a long time to zero, continuously making room for the advancement of high-frequency new patterns. The output of step S4 is a set of high-frequency common patterns across logs, whose direct effect is to supplement step S5 with composite structure patterns that were not discovered in the horizontal identification, improving the coverage and accuracy of key-value pair field extraction.

[0052] Step S5: Integrate the horizontal and vertical pattern mining results, and perform template fusion and deduplication based on similarity metric.

[0053] The purpose of step S5 is to use the bidirectional fixed writing pattern obtained in steps S3 and S4 to drive accurate key-value pair field extraction, and to integrate the parsing results into a structured key-value pair dictionary and a generalized log template through implicit variable post-processing and similarity aggregation, thereby eliminating redundant templates caused by differences in log samples.

[0054] First, candidate pattern integration and localization are performed: horizontal patterns are sorted in descending order by the number of key symbol tags they contain, and vertical patterns are sorted in descending order by global frequency. These are merged into a candidate pattern list. Then, a sliding window is used to locate the specific start and end positions of each candidate pattern within the current log's encoding sequence, ensuring that high-complexity, high-frequency patterns are prioritized for field extraction, thereby reducing ambiguity in subsequent conflict resolution. Next, pivot-point-driven key-value pair expansion is performed: for each candidate pattern's matching position in the log, using... <equals>or <colon>Using the labels as the pivot point, the recognition key field is expanded to the left and the recognition value field to the right respectively; during the expansion process, adjacent blank labels on both sides of the pivot are skipped, and the expansion stops when a boundary symbol or an area already marked and occupied is encountered to the left, while the expansion to the right supports the recognition of values ​​enclosed in quotes and brackets; for <colon>The core algorithm performs additional detection and exclusion of IP address and port formats to prevent misidentification. All identified key-value pair candidates are greedily prioritized based on the number of key symbol tags they contain. A bit array is used for conflict detection to ensure that each tag position is covered by at most one key-value pair, eliminating ambiguity in field boundaries caused by overlapping patterns. After extraction based on known patterns, the entire log's encoded sequence is re-executed with a core extension search to capture key-value pair structures not covered by any horizontal or vertical patterns, further improving the recall rate of field extraction. Subsequently, implicit variable identification is performed on the initially generated template. Implicit variables that have not yet been parameterized are identified in the template in the following priority order: URL, file path, IP address and port pairing, individual IPv4 / IPv6 addresses, UUID, and pure numbers. These variables are replaced with numbered named type tags, and the identified original values ​​are written into the parsing result dictionary. Finally, parameterized templates are generated based on the determined key-value pair replacement range, and the entire batch of templates is hierarchically aggregated according to structural similarity. Templates with similarity exceeding a preset threshold are merged into the same cluster, and the template with the widest coverage is used as the representative template. Redundancy is eliminated and a hierarchical index is built to support subsequent fast matching.

[0055] For large-scale log batch processing scenarios, this invention also provides a hybrid parsing scheduling framework: the input logs are divided into segments according to a preset size. For the first segment, if the system already has an initial template library, it will first attempt to perform fast matching and parsing based on the template. Logs that fail to match will be transferred to the full parsing process in steps S1 to S5 above, and the new templates generated will be added to the runtime template library. For subsequent segments, template matching will be performed first using the runtime template library. Unmatched logs will enter a waiting queue. When the waiting queue accumulates to a preset threshold, the full parsing process will be triggered for batch processing, and the generated new templates will be added to the runtime template library. Then the waiting queue will be cleared. The framework supports multi-process parallel scheduling. Different batches are allocated to independent processes for parallel processing through a process pool, and the main process can realize real-time aggregation and display of the global parsing progress by using a shared progress dictionary, which effectively improves the throughput of large-scale log processing.

[0056] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0057] To verify the overall technical effectiveness of the log template extraction method based on fixed writing pattern mining proposed in this invention, this invention conducts comparative tests with existing mainstream log parsing algorithms based on multiple publicly available real log datasets. The experiments use six evaluation metrics: grouping accuracy F1 score, template accuracy F1 score, grouping accuracy, parsing accuracy, template precision, and template recall. Fifteen mainstream log parsing methods in the industry, including Drain, Spell, Logram, and UniParser, are used as baseline control schemes. Detailed experimental results can be found in [link to relevant documentation]. Figures 5 to 8 .

[0058] Figure 5 The performance comparison results of various parsing methods in terms of grouping accuracy F1 value are shown. The grouping accuracy F1 value reflects the clustering grouping ability of the parser after filtering noisy logs and can be used to evaluate the stability of the algorithm in recognizing the main structure of the log. Experimental results show that the grouping accuracy F1 value of the method in this embodiment is 0.75, ranking second among 15 compared algorithms. The index value is close to that of the IPLoM algorithm (0.77), and the overall performance is better than many mainstream parsing methods such as Drain, UniParser, Logram, AEL, LSH, LFA, and SHISO. At the same time, it has certain performance advantages compared with traditional parsing algorithms such as LogCluster, LogSig, SLCT, and LogMine. The experimental results show that the technical solution of this embodiment, which combines structural semantic tag encoding and bidirectional fixed writing pattern mining, can effectively complete log clustering and merging without the need for manual pre-setting rules, and has good log grouping ability.

[0059] Figure 6 This section presents a comparison of the template accuracy F1 score across various parsing methods. The template accuracy F1 score, building upon the log grouping effect, further measures the precision of the algorithm's template extraction and is a crucial evaluation indicator of log template extraction quality. Experimental results show that the template accuracy F1 score of the method in this embodiment is 0.40, ranking first among all 15 compared algorithms, higher than the second-ranked Logram algorithm (0.29) and the remaining baseline methods. Compared to the traditional algorithm LogSig (0.04) and the commonly used mainstream algorithm Drain (0.22), the template accuracy F1 score of the method in this embodiment is significantly improved. This result demonstrates that the technical architecture of this embodiment, employing a combination of horizontal greedy fixed writing pattern recognition and vertical light-heavy separation pattern enhancement, can effectively optimize the refined extraction effect of log templates and effectively improve the problem of low template extraction accuracy in existing parsing methods.

[0060] Figure 7 The test scores and overall rankings of each parsing algorithm across six evaluation metrics are summarized in tabular form. Overall test data shows that the method of this invention ranks first in four metrics: grouping accuracy (F1 score), template precision, template recall, and template accuracy (F1 score). It also maintains a good level in both grouping accuracy and parsing accuracy, demonstrating balanced overall performance. Compared to existing parsing methods, this invention performs better in the template refinement evaluation metric, indicating that this method can not only effectively complete log grouping and clustering but also reconstruct the log structured template relatively completely and accurately, balancing the accuracy and completeness of template extraction.

[0061] Figure 8 The performance comparison results of each log parser under the parsing accuracy metric are shown, and the performance of each method on 6 test datasets are summarized. The parsing accuracy of the method in this embodiment is 0.949, which is the best among all the evaluated methods. The method in this embodiment performs stably on different datasets, with significant advantages on datasets such as Apache, Zookeeper, and HealthApp, and is basically on par with other parsers on other datasets, demonstrating balanced performance across scenarios.

[0062] The above experimental results verify that the overall technical solution formed by standardized tag encoding, bidirectional pattern mining, and similarity aggregation deduplication in the embodiments of the present invention has certain comprehensive performance advantages over the existing technology in terms of log parsing accuracy and structured output quality.

[0063] (1) Experiment on the impact of lateral mode on compression ratio and computational complexity; To verify the actual contribution of the lateral compression mechanism to the overall computational efficiency of the system, this embodiment of the invention conducted a comparative experiment on 14 real system log datasets from the Loghub 2.0 public benchmark set, with 2000 logs from each dataset used for testing. The experiment counted the total number of insertion operations in the vertical light-heavy separation structure with and without lateral compression enabled, and used the difference between the two sets of data to measure the computational reduction effect of lateral compression. At the same time, the ratio of the average sequence length before and after compression was recorded as the compression ratio.

[0064] like Figure 9 As shown, the results indicate that the average length of the log sequence after horizontal compression is 0.7797 compared to that before compression, meaning the average sequence length is shortened by approximately 22%. Regarding vertical computational overhead, the reduction rate of insert operations for the 14 datasets ranged from a minimum of 0.3% (Thunderbird) to a maximum of 90.7% (HDFS), with an average reduction rate of 35.1%. The reduction rates were particularly significant for datasets with numerous repetitive structures, such as HDFS, Proxifier, OpenStack, and BGL, reaching 90.7%, 74.2%, 50.5%, and 47.2%, respectively. This demonstrates that horizontal compression can effectively identify and fold repetitive subsequences within logs, thereby significantly reducing the computational cost of the vertical statistical stage without losing structural information.

[0065] (2) Comparison between the light and heavy separation structure and sliding window statistics; To verify the accuracy advantage of the light-heavy separation structure used in the embodiments of the present invention in high-frequency pattern recognition tasks, the experiment systematically compared it with the sliding window statistical method with fixed window sizes of 100 and 500. The precision, recall and F1 score between the high-frequency pattern set output by each method and the manually labeled real labels were used as evaluation indicators. At the same time, memory usage and processing time were statistically analyzed.

[0066] Under the same test conditions of 2000 log entries, the light-heavy separation structure achieved full marks in precision, recall, and F1 score (mean 1.000) on all 14 datasets, meaning that its output high-frequency pattern set completely matches the true labels, with no false positives or false negatives. In contrast, the sliding window method (window size 100 and 500) had a mean recall of only 0.390 and 0.573, and a mean F1 score of 0.512 and 0.684, respectively. It showed significant missed detections of high-frequency patterns on multiple datasets, especially performing poorly on datasets with a large number of pattern categories, such as BGL, Hadoop, HealthApp, and Thunderbird.

[0067] The above results show that the light-heavy separation structure can accurately maintain the frequency of all sliding window patterns within a limited processing time, and its recognition accuracy is significantly higher than that of the fixed window statistical method. It is a reasonable selection basis for the vertical pattern statistical stage of the present invention.

[0068] Table 1. Comparison of results between the light and heavy separation structure and the sliding window statistics. (3) Template stability indicators under different log types; Template stability is a core metric for evaluating the output quality of a log parser, directly impacting the automated operation and long-term reliability of downstream analysis tasks. To verify the template stability performance of the parser in this embodiment, this experiment conducts comparative tests based on the Loghub 2.0 public benchmark dataset. Six typical log datasets—Apache, Zookeeper, Hadoop, HealthApp, Spark, and Thunderbird—are selected, with parsing accuracy as the core quantitative evaluation metric. Performance is compared with six mainstream log parsing algorithms: Drain, Spell, Loogram, UniParser, LogMine, and AEL. Parsing accuracy represents the proportion of logs that correctly match the standard template out of the total number of logs. A higher value indicates a higher degree of matching between the generated template and the actual template, and better template output stability. All datasets in this experiment use manually annotated real results provided by Loghub 2.0 as the evaluation benchmark to ensure the objectivity and accuracy of the experimental evaluation.

[0069] The method proposed in this embodiment of the invention achieves an average parsing accuracy of 0.949 on six test datasets, outperforming all compared algorithms. The average parsing accuracies of the mainstream algorithms are as follows: Drain and UniParser 0.929, AEL 0.867, Spell 0.855, LogMine 0.790, and Locogram 0.432. In single-dataset test scenarios, the method of this embodiment achieves a perfect parsing accuracy of 1.000 on the Apache dataset, and parsing accuracies of 0.992 and 0.955 on the Zookeeper and Thunderbird datasets, respectively. On the HealthApp dataset, the method of this embodiment achieves a parsing accuracy of 0.897, showing a significant advantage over the best-performing compared algorithm UniParser (0.780) in this scenario. On the Hadoop dataset, the method of this embodiment achieves a parsing accuracy of 0.945, only a slight difference of 0.003 from the best algorithm Drain's accuracy of 0.948, indicating essentially equal performance. In summary, the pattern recognition mechanism of horizontal greedy iteration combined with vertical light and heavy separation adopted in the embodiments of the present invention can adapt to multiple types of log data, effectively improve the accuracy of log template parsing, and stably output parsing results that are highly consistent with the real templates. It has outstanding cross-scenario generalization ability and template stability.

[0070] (4) Real-time streaming processing latency and throughput test.

[0071] To evaluate the latency and throughput performance of the streaming online processing framework implemented in this embodiment of the invention in real-time scenarios, experiments were conducted to simulate a real streaming scenario where logs arrive one by one and processing is triggered in batches (chunks). The end-to-end latency and throughput of the complete processing flow were measured.

[0072] First, with the BGL dataset fixed, performance comparison experiments were conducted with five batch sizes: 10, 25, 50, 100, and 200. The throughput of all five experiments remained stable between 3106 and 3237 records per second, with a median latency between 0.185 and 0.233 milliseconds and a high quantile latency between 0.867 and 1.291 milliseconds. The experimental results show that the system throughput and low quantile latency are insensitive to changes in batch size, and the overall processing performance exhibits excellent stability. Subsequently, with a fixed batch size of 100, traversal tests were performed on all 14 datasets. The throughput of each dataset ranged from 1279 records per second (OpenStack) to 6538 records per second (HPC), with an average throughput of 2597 records per second for the entire dataset. The median latency was 0.456 milliseconds, and the high quantile latency was 0.689 milliseconds. The high quantile latency for all datasets did not exceed 1.3 milliseconds.

[0073] Experimental results demonstrate that the embodiments of the present invention can stably maintain sub-millisecond median latency and low-millisecond tail latency in streaming processing scenarios, which can fully meet the running and processing requirements of real-time log analysis tasks and has excellent real-time processing capabilities.

[0074] Table 2. Throughput and average processing latency at different batch sizes Table 3. Throughput and average processing latency for different datasets with a batch size of 100 It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.< / colon> < / colon> < / equals> < / content> < / colon> < / colon> < / equals> < / dot> < / hyphen> < / underscore> < / doublespace> < / space> < / colon> < / dot> < / rbracket> < / rbrace> < / rparen> < / equals> < / space> < / space> < / doublespace> < / space> < / tab> < / space> < / space> < / content> < / content> < / slash> < / quote> < / space> < / comma> < / colon> < / equals> < / rparen> < / lparen>

Claims

1. A log template extraction method based on fixed writing pattern mining, characterized in that, include: The original log text is processed by replacing fixed-structure entities to obtain normalized log text containing structure entity placeholders; The normalized log text is subjected to character-level structural semantic encoding to obtain an encoding sequence composed of structural semantic tags and content placeholders; The encoding sequence of a single log entry is subjected to horizontal fixed writing pattern recognition to obtain the horizontal fixed writing pattern that appears repeatedly within the log entry. The horizontal fixed writing pattern is used as a compression unit to perform horizontal compression on the encoded sequence in the streaming log, and cross-log candidate pattern mining is performed on the compressed sequence to obtain the vertical fixed writing pattern. Based on the horizontal fixed writing pattern and the vertical fixed writing pattern, the boundaries of structural fields are located in the log encoding sequence, a parameterized log template is generated, and the parameterized log template is deduplicated by similarity aggregation.

2. The method as described in claim 1, characterized in that, The fixed writing structure entity replacement process includes: compressing the original log text with whitespace characters; matching multiple timestamp formats according to a preset priority; replacing the matched timestamps with numbered timestamp placeholders; and establishing a mapping relationship between the timestamp placeholders and the original timestamp strings.

3. The method as described in claim 1, characterized in that, The character-level structural semantic encoding includes: retaining timestamp placeholders as indivisible atomic units; replacing special characters with structural semantics with corresponding structural semantic tags; merging consecutive ordinary text character segments into content placeholders; and merging consecutive whitespace tags into whitespace structural tags of different granularities according to their quantity.

4. The method as described in claim 1, characterized in that, The horizontal fixed writing pattern recognition includes: counting the frequency of consecutive tag pairs in a single log encoding sequence using a preset length window; removing tag pairs whose starting tags belong to the invalid starting tag set; determining the candidate seed order according to frequency and first occurrence position; and performing greedy look-ahead extension on the candidate seeds to obtain candidate horizontal fixed writing patterns. The greedy look-ahead extension includes: recording all starting positions of the candidate seed in the encoded sequence; counting the frequency of occurrence of the successor label at the current position; selecting the successor label that meets the frequency condition and is optimally sorted and appending it to the candidate seed; updating the set of starting positions that can still match the extended sequence; and stopping the extension when there is no successor label that meets the condition or when the maximum length is reached.

5. The method as described in claim 1, characterized in that, The generation of the parameterized log template includes: using relational label or structure separator label as the pivot point, performing field boundary expansion on both sides of the candidate pattern matching position; resolving bit conflicts for the identified key-value pair candidates; performing supplementary pivot scanning on log intervals not covered by candidate patterns; performing typed replacement on implicit variables; and generating a parameterized log template based on the final replacement range.

6. The method as described in claim 5, characterized in that, The implicit variables include Uniform Resource Locators (URLs), file paths, network address and port combinations, network addresses, unique identifiers, and plain numeric strings; the implicit variables are identified and replaced with numbered typed placeholders according to a preset priority. The similarity aggregation deduplication includes: calculating the edit distance or encoding sequence overlap rate between parameterized log templates; classifying parameterized log templates with similarity reaching a preset threshold into the same template cluster; selecting the parameterized log template with the largest coverage in the template cluster as the representative template; and establishing a hierarchical index between the representative template and the corresponding template cluster.

7. A log template extraction system based on fixed writing pattern mining, characterized in that, include: The structure entity replacement module is used to perform fixed-writing structure entity replacement processing on the original log text to obtain normalized log text containing structure entity placeholders. The character-level structural semantic encoding module is used to perform character-level structural semantic encoding on the normalized log text to obtain an encoding sequence composed of structural semantic tags and content placeholders; The horizontal fixed writing pattern recognition module is used to recognize the horizontal fixed writing pattern of the encoded sequence of a single log and obtain the horizontal fixed writing pattern that appears repeatedly within the log. The vertical fixed writing pattern mining module is used to use the horizontal fixed writing pattern as a compression unit to perform horizontal compression on the encoded sequence in the streaming log, and to perform cross-log candidate pattern mining on the compressed sequence to obtain the vertical fixed writing pattern. The log template generation module is used to locate the structural field boundaries in the log encoding sequence based on the horizontal fixed writing pattern and the vertical fixed writing pattern, generate a parameterized log template, and perform similarity aggregation to remove duplicates from the parameterized log template.

8. The system as described in claim 7, characterized in that, The structural entity replacement module is used to compress the original log text with whitespace characters, match multiple timestamp formats according to a preset priority, replace the matched timestamps with numbered timestamp placeholders, and establish a mapping relationship between the timestamp placeholders and the original timestamp strings. The character-level structural semantic encoding module is used to retain the timestamp placeholders as indivisible atomic units, replace special characters with structural semantics with corresponding structural semantic tags, merge continuous ordinary text character segments into content placeholders, and merge continuous whitespace tags into whitespace structural tags of different granularities according to their quantity. The horizontal fixed writing pattern recognition module is used to count the frequency of continuous tag pairs in a single log encoding sequence with a preset length window, remove tag pairs whose starting tags belong to the invalid starting tag set, determine the candidate seed order according to frequency and first occurrence position, and perform greedy look-ahead extension on the candidate seeds to obtain candidate horizontal fixed writing patterns.

9. A method for reinforcing a vertically fixed writing pattern in streaming logs, characterized in that, include: Obtain the log encoding sequence and its corresponding horizontal fixed writing pattern; After sorting the horizontal fixed writing patterns by length, perform exact matching in the log encoding sequence, and replace the matched tag segments with pattern placeholders to obtain the compressed encoding sequence; The compressed encoding sequence is enumerated using a sliding window, and the pattern placeholders in the window are expanded into the corresponding original encoding sequences to obtain candidate vertical patterns. The candidate longitudinal patterns are validated, and the validated candidate longitudinal patterns are written into the light and heavy separation frequency statistics structure. Based on the frequency results in the frequency statistics structure for separating light and heavy elements, output the vertical fixed writing pattern that meets the support condition.

10. The method as described in claim 9, characterized in that, The light-heavy separation frequency statistics structure includes a heavy mass bucket and a light mass bucket; the heavy mass bucket is used to store the exact count and defense value of candidate longitudinal patterns; the light mass bucket is used to store the approximate frequency of candidate longitudinal patterns that have not entered the heavy mass bucket; when the target bucket of the heavy mass bucket is full, it is determined whether to replace the existing patterns in the target bucket based on the defense value, approximate frequency and exact count, and the defense value in the heavy mass bucket is attenuated in batches.

Images