Log analysis method based on size model cooperation and computer readable storage medium

Abstract

The application discloses a log analysis method and a computer readable storage medium based on a large model and a small model, and belongs to the technical field of log analysis. The method comprises the following steps: for each token subsequence corresponding to a log, the following steps are performed: if the token subsequence matches a cached log template successfully, a matching result and a parameter position are output, and the analysis is completed; otherwise, a first token is selected; a small language model is used to analyze template tokens corresponding to the current token, and a routing model is used to predict a routing probability that needs to be corrected; if the routing probability is greater than a routing threshold, a large language model is used to analyze template tokens corresponding to the target token, and a template prefix is added; otherwise, template tokens predicted by the small language model are added to the template prefix; if the analysis is not completed, the next token is selected for analysis; otherwise, a current template prefix and a parameter position are output. The application can reduce the calling cost of the large language model in a large-scale log stream scene, and improve the log analysis accuracy.

Description

Technical Field

[0001] This invention belongs to the field of log parsing technology, and more specifically, relates to a log parsing method and computer-readable storage medium that uses a combination of large and small models. Background Technology

[0002] Log parsing is a fundamental step in intelligent operations and maintenance (O&M) and log analysis. Its goal is to transform raw log messages into a structured form, extracting static log templates and corresponding dynamic parameters. For example, for log messages containing runtime variables such as IP addresses, paths, task numbers, and process IDs, log parsing methods typically replace the changing parts with wildcards or placeholders while retaining the fixed text describing system events. The parsed log templates can then be used for downstream tasks such as anomaly detection, fault diagnosis, root cause analysis, log compression, and security auditing. Existing research defines log parsing as the process of extracting static templates and dynamic parameters from free text logs, indicating that this task is a prerequisite for log analysis.

[0003] Existing log parsing methods can be broadly categorized into syntax rule-based methods, semantic model-based methods, large language model-based methods, and collaborative methods of large and small language models that have emerged in recent years.

[0004] Log parsing methods based on grammatical rules or heuristic features, such as Drain, AEL, Brain, and Loogram, typically group logs with similar structures into template groups based on the number of log tokens, token positions, term frequencies, fixed-depth parse trees, frequent patterns, or manually defined heuristic rules, and then extract fixed text as log templates. These methods are simple to implement and relatively efficient, but their parsing performance depends on pre-defined rules, human experience, and specific log format assumptions. When log formats are complex, parameter lengths are variable, log templates are constantly evolving, or there are significant differences in log structures across systems, rules or heuristic features often fail to reliably cover the actual situation, leading to inaccurate template extraction or incorrect log grouping. Existing research has also pointed out that traditional grammatical log parsers, which typically rely on manually designed rules or heuristic features, have limited adaptability to diverse and evolving logs.

[0005] Log parsing methods based on semantic models or small language models, such as UniParser, LogPPT, and Unleash, typically model log parsing as a token classification task or a template / parameter recognition task. They train or fine-tune small neural network models or pre-trained language models using labeled log data, enabling the model to learn the semantic differences between static text and dynamic parameters in the logs. Compared to pure rule-based methods, these methods have stronger semantic expressive power and generally lower inference costs than large language models, making them suitable for deployment in resource-constrained or high-throughput scenarios. However, these methods usually require a certain scale of labeled log data for training or fine-tuning, but log labeling is costly, and different systems have significantly different log formats, making it difficult for existing labeled data to cover new systems, new versions, or new templates. Furthermore, small models are limited by their capacity, and their parsing accuracy tends to decrease when handling complex semantics, rare templates, cross-system migration, and continuously evolving logs.

[0006] Log parsing methods based on large language models, such as LogParser-LLM, DivLog, LILAC, and LogBatcher, have great potential in complex logs, cross-system logs, and low-label data scenarios due to the strong natural language understanding and code-related knowledge of large language models. This is because large language models can identify dynamic variables and static templates in logs with few samples or under cue-driven conditions. However, large language models are not specifically trained for log parsing tasks, and directly calling them can lead to unstable output, inconsistent template granularity, and inconsistent parameter extraction. Furthermore, real-world software systems continuously generate large-scale logs; calling large language models for each log entry would incur high inference costs, network latency, and computational resource overhead, hindering practical deployment. LILAC summarizes these problems as: large language models lacking specialized log parsing capabilities, inconsistent output, and the significant overhead of directly using large language models.

[0007] Log parsing methods that combine large and small language models, such as CSLParser, fine-tune the small language model by selecting training samples through multi-layer feature clustering and diversified sampling in the offline stage. In the online stage, the small language model first matches existing templates using a cache. When the cache is not hit, the small language model performs the parsing and identifies difficult samples that the small language model cannot handle according to preset rules. Then, the large language model is selectively invoked for correction. Simultaneously, this method uses a dynamic template update mechanism to merge similar templates based on structural and semantic information to reduce redundant templates and improve parsing accuracy. This method introduces the idea of ​​large and small model collaboration, allowing the small model to undertake most of the log parsing tasks, while the large model only handles difficult samples, thus balancing efficiency and accuracy. However, in existing log parsing methods that combine large and small language models, the large model's invocation mainly relies on preset rules, such as structural complexity, keywords, and confidence thresholds. These rules typically apply to the entire log or the parsing result level, making it difficult to capture finer-grained local errors during the parsing process. Furthermore, its collaborative granularity is mainly sample-level or result-level correction, rather than real-time routing for model divergences of each token during log template generation; when a small model makes a minor error on a local token that affects the template structure, the rule-level triggering mechanism may have difficulty identifying it in a timely manner. Summary of the Invention

[0008] To address the shortcomings and improvement needs of existing technologies, this invention provides a log parsing method and computer-readable storage medium that coordinates large and small models. The purpose is to reduce the cost of calling large language models and improve the accuracy of log parsing in large-scale log stream scenarios.

[0009] To achieve the above objectives, according to one aspect of the present invention, a log parsing method based on a size-model collaboration is provided, comprising: The log text to be parsed is preprocessed into a token sequence, and the positional mapping relationship between each token and the original log text is recorded. For each log entry, the corresponding token subsequence is extracted, and the log parsing steps are performed. The log parsing steps include: S1: Match the current token subsequence with the cached log template. If the match is successful, use the current log template as the current log template and output the corresponding template string and parameter position. The parsing of the current log ends. Otherwise, select the first token in the current token subsequence as the target token, initialize the template prefix to empty, and go to S2. S2: Use a small language model to parse the template token corresponding to the target token; S3: Use the trained routing model to predict the routing probability that needs to be corrected for the template token parsed by the small language model. If the routing probability is greater than the preset routing threshold, proceed to S4; otherwise, add a template prefix to the template token output by the small language model and proceed to S5; the routing model is a neural network model. S4: Use a large language model to parse the template token corresponding to the target token and add a template prefix; S5: If the current token subsequence has been processed, proceed to S6; otherwise, select the next token as the target token and proceed to S2. S6: Use the current template prefix as the log template for the current log, and output the corresponding template string and parameter positions. The parsing of the current log ends.

[0010] Furthermore, S1 also includes: If the current token subsequence does not match the cached log template, the similarity between the current token subsequence and each cached log template is calculated as the candidate score of each log template, and the log template with the candidate score higher than the preset candidate score is taken as the fuzzy hit result. Furthermore, in S4, when using the large language model to parse the template token corresponding to the target token, the input of the large language model includes: the original log, the current template prefix, the template token parsed by the small language model, the fuzzy hit result, and the prompt information.

[0011] Furthermore, in S2, the small language model outputs the features of the template token while parsing the template token corresponding to the target token; Furthermore, in S3, when using the pre-trained routing model to predict the route probability that needs to be corrected for the template token parsed by the small language model, the input of the routing model includes: the template token parsed by the small language model, the features of the template token, the log template with the highest candidate score in the fuzzy hit results, and the number of log templates in the fuzzy hit results.

[0012] Furthermore, the training methods for the routing model include: Collect historical log text, preprocess it into a token sequence, and then use a small language model and a large language model to parse the template token corresponding to each token. Compare the differences in parsing each token by the small language model and the large language model. If the parsing results of the small language model and the large language model are consistent for the same token, then the corresponding token position is marked as no routing required, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, but the difference does not affect the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as neutral difference, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, and the difference affects the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as critical divergence, and the routing probability is marked as 1. A neural network model is trained using the labeled token sequence to obtain a trained routing model.

[0013] Furthermore, the log parsing results include: log template, template identifier, template string, parameter position, parsing source, cache hit identifier, routing information, and parsing confidence. Among them, the template identifier is used to uniquely identify the log template; the parsing source is used to mark whether each template token in the log template is parsed by the large language model or the small language model; the cache hit identifier is used to mark whether the log template hits the cache; the routing information is used to record the number of template tokens in the log template parsed by the large language model; and the parsing confidence is the weighted sum of the routing probabilities of each template token.

[0014] Furthermore, in S6, before using the current template prefix as the log template for the current log and outputting the corresponding parameter position, it also includes: merging consecutive template tokens identified as parameters in the current template prefix into a single parameter placeholder and updating the parameter position information.

[0015] Furthermore, log templates are stored in the cache in the form of a prefix tree, and a template entity index table is built in the cache; The template entity index table is used to store the index information of each log template; the index information includes: template string, number of hits, number of conflicts, risk score, template status, and associated logs.

[0016] Furthermore, a template index table is also established in the cache; the template index table is used to record the template identifier and the corresponding template string of each template in the cache; Furthermore, in S1, the current token subsequence is matched against the cached log template, including: S11: Search the template index table according to the string corresponding to the current log. If there is a template string in the template index table that is completely consistent with the string, the match is successful and proceed to S13; otherwise, proceed to S12. S12: Query the current token subsequence in the prefix tree. If the query is successful, the match is considered successful and proceed to S13; otherwise, the match is considered unsuccessful and proceed to S13. S13: Matching ends.

[0017] Furthermore, S6 also includes: performing a template evolution operation based on the parsing result; the template evolution operation includes: If the current parsing result does not hit the cache, then perform the following operations: If the log template corresponding to the current log is the same as the cached log template t1, then update the index information of log template t1; If the current log template is inconsistent with the cached log template, but its similarity with the cached log template t2 is higher than the preset similarity threshold, and the difference between the two is caused by the change of parameter value, then the difference position in the log template t2 is generalized to parameter placeholders, and the index information of the generalized log template t2 is updated to realize the merging of log templates. If the log template corresponding to the current log is inconsistent with both the cached log templates and cannot be merged with the cached log templates, then the log template corresponding to the current log is inserted as a new log template into the prefix tree, and the index information of the log template corresponding to the current log is generated and inserted into the template entity index table; in the generated index information, the hit count is 0, the conflict count is 0, the template status is candidate template, and the associated log is the current log. Insert the template string and template identifier corresponding to the log template into the template index table; If the current parsing result hits the cache, then perform the following operations: If a conflict is detected in the hit results, the conflict count of the hit log template is incremented by 1 and its risk score is increased; otherwise, the hit count of the hit log template is incremented by 1 and its risk score is decreased. If the number of hits of a candidate template increases continuously, and the number of consecutive increases reaches a preset stable threshold, then the template status of the log template will be updated to a stable template, and its risk score will be reduced. If the number of hits of a stable template increases continuously, and the number of consecutive increases reaches a preset support threshold, then the template status of the stable template is updated to frozen template, and its risk score is reduced; the status of the frozen template is no longer updated. If the number of conflicts of a stable template increases and the increase reaches a preset backoff threshold, the template status of the stable template will be updated to a candidate template and its risk score will be increased. If the number of conflicts of a candidate template reaches the preset risk threshold, the template status of the log template will be updated to a high-risk template and its risk score will be increased. If a high-risk template does not form multiple conflict patterns that can be split, and its conflict count increases continuously until the number of consecutive increases reaches a preset obsolescence threshold, then the template status of the log template will be updated to obsolescence and it will be removed from the cache. If the number of conflicts of a high-risk template reaches the preset threshold for splitting, the template status of the high-risk template will be updated to a template to be split, and the logs that conflict with the template to be split and the corresponding log templates will be recorded. If the recorded logs correspond to multiple incompatible log templates, the template to be split is split into multiple new candidate templates, index information of each candidate template is generated and inserted into the template entity index table, the template string and template identifier corresponding to each candidate template are inserted into the template index table, and the original template to be split is removed from the cache. If the recorded log does not yet correspond to multiple incompatible log templates, then the log template remains in a pending split state, and subsequent conflict logs continue to be recorded. Furthermore, in S1, when matching the current token subsequence with the cached log templates, log templates with lower risk scores have higher matching priority.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program that, when executed by a processor, implements the log parsing method for size-model collaboration provided by the present invention.

[0019] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention reduces model calls for duplicate logs by prioritizing template caching for matching. When the exact match in the cache fails, the corresponding template token is parsed token by token using a small language model. While the small language model is parsing, a neural routing model is used to predict the routing probability that the template token parsed by the small language model needs to be corrected. When the routing probability is high, i.e. the divergence between the large and small language models is large, the large language model is used for parsing. Through this parsing process, this invention achieves large and small model collaboration at the token granularity. In the collaboration process, the small language model undertakes most of the low-risk parsing tasks, reducing the overall inference cost. The large language model is only involved at key divergence positions by controlling the large language model through token-level neural routing, reducing the number of large language model calls and online latency. Furthermore, local correction is used to prevent the accumulation of small language model errors during template generation, thereby improving the parsing accuracy of complex logs and evolutionary logs.

[0020] (2) In the preferred embodiment of the present invention, when the exact matching cache fails, the present invention further searches for candidate templates to obtain fuzzy hit results, and uses the model hit results as one of the inputs of the large language model parsing, thereby improving the consistency of the output results of the large language model and alleviating the problem of unstable output of the large language model.

[0021] (3) In the preferred embodiment of the present invention, the present invention uses the fuzzy hit result as one of the inputs of the routing model to assist the routing model in predicting the corresponding routing probability, which can effectively improve the prediction accuracy of the routing probability. At the critical divergence position, the large language model is introduced in time to correct the parsing result to ensure the accuracy of the parsing result, and the small language model is used to parse at non-critical divergence positions to reduce the reasoning cost.

[0022] (4) In a preferred embodiment of the present invention, the present invention uses a large language model and a small language model to parse the same historical log token sequence, compares the parsing results of the two, identifies key divergence positions, and uses the comparison and identification results to label the historical token sequence for training the routing model, thereby ensuring the prediction accuracy of the routing model.

[0023] (5) In a preferred embodiment of the present invention, the present invention will also adaptively update the cached log template based on the parsing results and the divergence in the parsing process, thereby enabling the cached log template to adapt to the log to be parsed, improving the cache hit rate, reducing the number of times the model is called for parsing, and improving parsing efficiency while ensuring parsing accuracy. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a log parsing method for collaborative small and large models provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the cache index structure for log template caching provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the routing model training provided in an embodiment of the present invention.

[0027] Figure 4 The flowchart for collaborative parsing of large and small models provided in the embodiments of the present invention is shown.

[0028] Figure 5 This is a log template cache evolution state transition diagram provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0030] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0031] Existing methods for collaborative log parsing of large and small models can reduce the frequency of large model calls, but their collaborative triggering is mostly at the sample or rule level, making it difficult to detect and control fine-grained model divergences in real time during log template generation. To address this issue, this invention provides a collaborative log parsing method for large and small models and a computer-readable storage medium. It reduces model calls to duplicate logs by prioritizing template caching and matching. During log parsing, it coordinates the small and large language models at the token level. Specifically, the small language model undertakes most of the low-risk parsing tasks, reducing overall inference costs; token-level neural routing controls the large language model to intervene only at key divergence points, reducing the number of large language model calls and online latency; and local correction prevents the accumulation of small language model errors during template generation, improving the parsing accuracy of complex and evolutionary logs. Building upon this foundation, the present invention further incorporates model divergence information into cache updates, enabling a more accurate distinction between differences that should be generalized into parameters and semantic differences that should be retained as different templates, thereby reducing template over-generalization and template fragmentation issues. Through cache addition, merging, splitting, correction, and eviction mechanisms, the log parsing system's adaptability to software version changes, log format changes, and operating environment changes is improved.

[0032] This invention relates to the fields of computer software and artificial intelligence, and particularly to log parsing and structuring in intelligent operation and maintenance. More specifically, it relates to a method for parsing unstructured or semi-structured system logs into structured log templates and corresponding parameters by using neural network routing to perform token-by-token collaborative calls between small and large language models, and by adaptively evolving log template caching based on model parsing divergences.

[0033] This invention can be applied to the operation log processing scenarios of distributed systems, cloud computing platforms, operating systems, database systems, microservice systems, network devices, and various software systems. Specifically, it can be used as a pre-log parsing module for intelligent operation and maintenance tasks such as anomaly detection, fault diagnosis, root cause analysis, log compression, operation status monitoring, and security auditing. It can also be integrated into log analysis platforms, observability platforms, or automated operation and maintenance systems to efficiently and accurately extract templates and identify parameters from continuously generated large-scale log streams.

[0034] This invention provides a log parsing method that combines large and small language models, comprising stages such as log preprocessing, template cache matching, initial parsing of the small language model, token-level neural routing, local correction of the large language model, and final template generation. During log parsing, different parsing paths are formed based on cache hits and routing results. The main functions implemented in each stage are as follows: Log preprocessing includes: receiving raw log messages, performing log header recognition, log text extraction, word segmentation, special symbol processing, and variable format normalization on the raw logs to obtain a log text token sequence, and preserving the positional mapping relationship between the token sequence and the original logs. Log preprocessing provides a unified input for subsequent template matching, model parsing, and parameter extraction.

[0035] Template cache matching includes: retrieving cached log templates before model parsing; maintaining an adaptive log template cache, which stores parsed log templates, template identifiers, static tokens, parameter placeholders, representative log samples, hit frequency, confidence level, version information, and divergence statistics. Preferably, the log template cache adopts a hybrid index structure based on a prefix tree, including a prefix tree, a template entity index table, and a normalized template index table. When the current log arrives, the system first matches it with the cached template; if the cache hits, the corresponding log template and parameter sequence are directly output, and the template status is updated; if the cache misses, the model parsing process begins.

[0036] The initial parsing of the small language model includes: low-cost preliminary parsing of logs that have not been cached. Specifically, it receives the preprocessed log text token sequence, generates an initial log template or token tag sequence, and outputs intermediate information for each parsing position, such as confidence, candidate token probability, output entropy, hidden state, and context representation. Most parsing tasks for regular logs and low-risk tokens will be completed through the initial parsing of the small language model.

[0037] Token-level neural routing involves determining whether the parsing result of the small language model at the current position requires intervention from the large language model. Specifically, it integrates features such as the output probability of the small language model, output entropy, candidate token probability difference, current token type, generated template context, cached matching status, candidate template similarity, and historical divergence statistics to output the probability of a key parsing divergence at the current position. If this probability is lower than a preset threshold, the output of the small language model is accepted; if the probability is higher than the preset threshold, the current position is marked as a key divergence position, and local correction by the large language model is triggered.

[0038] The large language model's local correction involves handling key divergence points determined by the neural routing module. Specifically, it receives the current log, generated partial templates, the current output of the small language model, candidate cached templates, and parsing task hints, and corrects only the current token, current parameter boundaries, or current local template fragments. The correction result serves as the final parsing result for the current position and continues to be used as the context for subsequent template generation. This forms a token-by-token collaborative parsing process where the small language model is dominant and the large language model intervenes on demand.

[0039] The final template generation process includes summarizing the output of the small language model and the correction results of the large language model to generate the final log template and parameter sequence. As a preferred implementation, for tokens continuously identified as dynamic variables, the system merges them into a single parameter placeholder and extracts the corresponding parameter values ​​from the original log based on the position mapping relationship saved during the preprocessing stage. The final output includes the log template, template identifier, parameter sequence, parameter position, resolution confidence, and model divergence information.

[0040] Based on the above parsing process, this invention further proposes template cache evolution, updating the log template cache according to the parsing results and model divergence information; if the final template is consistent with an existing template, the hit frequency, confidence, and associated log samples of the template are updated; if the final template is similar to an existing template, and the difference is a parameter value change, the position is generalized to a parameter placeholder and the template is merged; if the final template is not similar to an existing template, or is similar but the difference is a static token or will change the semantics of the log event, it is determined that the final template cannot be merged with the existing template, and it is inserted into the cache as a new template; if a template has a long-term low frequency, low confidence, or continuous conflict, it is downgraded, corrected, merged, or deleted.

[0041] This invention can be implemented as an electronic device, a server cluster, or a cloud service. The electronic device includes a processor, memory, and a communication interface. The memory stores a log parsing program, template cache data, routing model parameters, and model call configuration. When the processor executes the program, it performs log preprocessing, cache matching, small language model parsing, token-level routing, large language model correction, template generation, and cache evolution operations. For production environments with large log volumes, cache matching and small language model parsing can be deployed locally or on edge nodes, while the large language model can be deployed on a high-performance server or cloud service, and the calling frequency of the large language model can be controlled through a routing module.

[0042] This invention can also be implemented in the form of a computer-readable storage medium. The storage medium stores a computer program that, when executed by a processor, causes the processor to perform the aforementioned log parsing method. The storage medium may be a disk, solid-state drive, flash memory, read-only memory, random access memory, or other media capable of storing program instructions.

[0043] In this implementation, the template cache prioritizes handling duplicate logs, the small language model handles most low-risk tokens, the neural routing module identifies key divergence points, the large language model only corrects key tokens or local template fragments, and the cache evolution module continuously maintains the template set based on stable hits, model corrections, and template conflicts. Therefore, this invention can reduce the number of large language model calls while ensuring log parsing accuracy and improving the system's adaptability to log template evolution.

[0044] The following is an example.

[0045] Example 1: A log parsing method that coordinates size models, such as Figure 1 As shown, it includes: The log text to be parsed is preprocessed into a token sequence, and the positional mapping relationship between each token and the original log text is recorded. For each log, the corresponding token subsequence is extracted, and the log parsing steps are performed.

[0046] In this embodiment, the preprocessing of the log text to be parsed into a token sequence specifically includes: first, identifying the log header fields and log text from the original log and preserving the correspondence between them; then, identifying the variable format of the log text, replacing common runtime variables with uniform placeholders, such as replacing IPv4 addresses with IP address placeholders, file paths with path placeholders, hexadecimal identifiers with hexadecimal placeholders, universal unique identifiers with unique identifier placeholders, and numeric fragments with numeric placeholders. For port numbers, task numbers, block numbers, container numbers, node numbers, timestamps, URLs, exception numbers, etc., appearing in the log, normalization can also be performed according to regular expression rules, lexical rules, or domain dictionaries. After normalization, the log text is segmented into a token sequence, and the positional mapping relationship between the tokens and the character ranges of the original log is preserved. This positional mapping relationship is used to subsequently recover parameter values ​​from the original log.

[0047] like Figure 1 As shown, in this embodiment, the log parsing steps include: S1: Match the current token subsequence with the cached log template. If the match is successful, use the current log template as the current log template and output the corresponding template string and parameter position. The parsing of the current log ends. Otherwise, select the first token in the current token subsequence as the target token, initialize the template prefix to empty, and go to S2. S2: Use a small language model to parse the template token corresponding to the target token; S3: Use the trained routing model to predict the routing probability that needs to be corrected for the template token parsed by the small language model. If the routing probability is greater than the preset routing threshold, proceed to S4; otherwise, add a template prefix to the template token output by the small language model and proceed to S5; the routing model is a neural network model. S4: Use a large language model to parse the template token corresponding to the target token and add a template prefix; S5: If the current token subsequence has been processed, proceed to S6; otherwise, select the next token as the target token and proceed to S2. S6: Use the current template prefix as the log template for the current log, and output the corresponding template string and parameter positions. The parsing of the current log ends.

[0048] Optionally, such as Figure 2 As shown, in this embodiment, the log template is stored in the cache in the form of a prefix tree, and a template entity index table is established in the cache; The template entity index table is used to store the index information of each log template; the index information includes: template string, number of hits, number of conflicts, risk score, template status and associated logs; A template index table is also established in the cache; the template index table is used to record the template identifier and the corresponding template string of each template in the cache.

[0049] In the log parsing step, S1 determines whether the log to be parsed can be directly parsed by a cached log template. In the cache, a prefix tree organizes log templates according to template token sequences, with each node representing a template token and the path between nodes representing a complete or partial log template. The template entity index table is used to read template entities based on template identifiers. Template entities include template strings, normalized templates, template sources, hit counts, large model correction counts, conflict counts, stable hit counts, risk scores, template status, representative log samples, last update time, and metadata. The normalized template index table establishes a mapping between normalized template strings and template identifiers to support exact matching and duplicate template detection.

[0050] When performing template matching, S1 prioritizes exact matching. Specifically, the log to be parsed is normalized to obtain a normalized log string, and the system searches the normalized template index table for a completely matching template. If a completely matching template exists and it is not marked as obsolete, the cache is considered hit, and the system directly outputs the template, extracting parameter values ​​based on the correspondence between the original log and the template placeholders. In this case, there is no need to call the small language model and the large language model.

[0051] If an exact match fails, the model needs to be invoked for parsing. To further improve the parsing accuracy of the model, as a preferred implementation, this embodiment also includes: if an exact match fails, performing a candidate template retrieval. During candidate template retrieval, the similarity between the current token subsequence and each cached log template is calculated as the candidate score for each log template, and log templates with candidate scores higher than a preset candidate score are taken as fuzzy hit results. Specifically, when calculating the similarity between the current token subsequence and each cached log template, factors such as token set overlap, string similarity, template length similarity, placeholder position similarity, and log header field consistency between the current token subsequence and the cached log templates can be considered. Preferably, this embodiment performs a weighted sum of token set overlap and string similarity to obtain a basic similarity; then, the candidate score is corrected based on the log template's risk score, historical hit count, template status, and log header metadata consistency. For templates with high risk, many conflicts, or that have been abandoned, the system lowers their candidate score; for templates with many hits and consistent metadata, the system raises their candidate score.

[0052] In this embodiment, S2 in the log parsing step performs token-by-token parsing on the token subsequence corresponding to the current log when a cache miss occurs, generating an initial log template. Optionally, in this embodiment, the input to the small language model includes: normalized log text, token sequence, the token to be parsed, optional context log and log header context, and the output includes: initial template, template token sequence, and confidence information for each template token. The initial template output by the small language model includes static tokens and parameter placeholders. For example, for the log "Failed to connect to port 8080 of node 10.0.0.1", the small language model can output "Connected to node..." <ip>port <num>fail".

[0053] In a preferred implementation, the output of the small language model includes not only the initial template but also the features of the template token. The features of the template token include the template token parsed by the small language model, the distribution of candidate tokens, the probability p1 of the highest candidate token, the probability p2 of the second highest candidate token, the difference between p1 and p2, the output entropy, the hidden state vector, and the current prefix context. The features of the template token output by the small language model are input into the routing model to assist in predicting the routing probability that needs to be corrected for the template token currently parsed by the small language model.

[0054] Accordingly, in S3, when using the trained routing model to predict the route probability that needs to be corrected for the template token parsed by the small language model, the input of the routing model includes: the template token parsed by the small language model, the features of the template token, the log template with the highest candidate score in the fuzzy hit results, and the number of log templates in the fuzzy hit results.

[0055] In this embodiment, the routing model is implemented using a lightweight neural network, such as a multilayer perceptron, a feedforward neural network, or a lightweight Transformer encoder. Preferably, the routing model uses a feedforward neural network consisting of an input projection layer, several fully connected hidden layers, and a binary classification output layer. The input projection layer encodes the template token parsed by the small language model, the features of the template token, the log template with the highest candidate score, and the number of log templates in the fuzzy hit results, and concatenates the encoding results before inputting them into the fully connected hidden layer. The binary classification output layer outputs the routing probability that the current template token needs to be corrected by the large language model. The routing model can be obtained through offline training, and its training method is as follows: Figure 3 As shown, it includes: Collect historical log text, preprocess it into a token sequence, and then use a small language model and a large language model to parse the template token corresponding to each token. Compare the differences in parsing each token by the small language model and the large language model. If the parsing results of the small language model and the large language model are consistent for the same token, then the corresponding token position is marked as no routing required, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, but the difference does not affect the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as neutral difference, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, and the difference affects the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as critical divergence, and the routing probability is marked as 1. A neural network model is trained using the labeled token sequence to obtain a trained routing model. The trained routing model identifies key divergence tokens during online parsing.

[0056] Based on the trained routing model, the parsing process for the collaboration between the large and small models is as follows: Figure 4 As shown.

[0057] When the template token, its features, the log template with the highest candidate score in the fuzzy hit results, and the number of log templates in the fuzzy hit results are input into the routing model, the routing model will output the routing probability that the template token parsed by the small language model needs to be corrected. If the routing probability is greater than the preset routing threshold, it indicates that the current token may cause template structure errors, parameter boundary errors, or log event semantic classification errors, and is a high-risk divergence token, which needs to be corrected locally by calling the large language model.

[0058] The routing threshold can be a globally fixed threshold, or different thresholds can be set according to the log source system, log format, component type, or log level.

[0059] S4 corrects high-disagreement-risk tokens using a large language model. When parsing the template token corresponding to the target token using the large language model, the input of the large language model includes: the original log, the current template prefix, the template token parsed by the small language model, the fuzzy hit result, and the prompt information.

[0060] The large language model only provides correction suggestions for the current token and current parameter boundaries, without having to regenerate a complete template for the entire log. If the large language model returns a valid correction result, the system uses the output of the large language model as the final token for the current position; if the large language model does not return a valid result, the system retains the output of the small language model. In this way, the large language model intervenes only at critical locations, which can reduce the cost of invocation and inference latency.

[0061] Finally, the template tokens generated by the small language model and the large language model are concatenated in sequence to form the log template corresponding to the current log. In this embodiment, consecutive parameter tokens in the log template are merged before outputting the parsing results to avoid generating multiple adjacent placeholders.

[0062] Optionally, in this embodiment, the log parsing result includes: log template, template identifier, template string, parameter position, parsing source, cache hit identifier, routing information, and parsing confidence. Among them, the template identifier is used to uniquely identify the log template; the parsing source is used to mark whether each template token in the log template is parsed by the large language model or the small language model; the cache hit identifier is used to mark whether the log template hits the cache; the routing information is used to record the number of template tokens in the log template parsed by the large language model; and the parsing confidence is the weighted sum of the routing probabilities of each template token.

[0063] As a preferred implementation method, such as Figure 1 As shown, in this embodiment, after each parsing, the cached log template is adaptively updated based on the parsing results and model divergence information. The state transition relationship of the log template is as follows: Figure 5 As shown, based on signals such as the final template, the initial template of the small language model, the cached matching results, whether the large language model is involved, whether the template has changed, candidate template similarity, number of conflicts, and number of stable hits, the system determines whether to perform addition, promotion, generalization, degradation, splitting, or discarding operations on the cache. Accordingly, S6 also includes: performing template evolution operations based on the parsing results; the template evolution operations include: If the current parsing result does not hit the cache, perform the following operations: If the log template corresponding to the current log is the same as the cached log template t1, then update the index information of log template t1; If the current log template is inconsistent with the cached log template, but its similarity with the cached log template t2 is higher than the preset similarity threshold, and the difference between the two is caused by the change of parameter value, then the difference position in the log template t2 is generalized to parameter placeholders, and the index information of the generalized log template t2 is updated to realize the merging of log templates. If the log template corresponding to the current log is inconsistent with the cached log template and cannot be merged with the cached log template, then the log template corresponding to the current log is inserted into the prefix tree as a new log template, and the index information of the log template corresponding to the current log is generated and inserted into the template entity index table. In the generated index information, the hit count is 0, the conflict count is 0, the template status is candidate template, and the associated log is the current log. Insert the template string and template identifier corresponding to the log template into the template index table; If the current parsing result hits the cache, then perform the following operations: If a conflict is detected in the hit results, the conflict count of the hit log template is incremented by 1 and its risk score is increased; otherwise, the hit count of the hit log template is incremented by 1 and its risk score is decreased. If the number of hits of a candidate template increases continuously, and the number of consecutive increases reaches a preset stable threshold, then the template status of the log template will be updated to a stable template, and its risk score will be reduced. If the number of hits of a stable template increases continuously, and the number of consecutive increases reaches a preset support threshold, then the template status of the stable template will be updated to frozen template, and its risk score will be reduced; the status of the frozen template will no longer be updated. If the number of conflicts of a stable template increases and the increase reaches a preset backoff threshold, the template status of the stable template will be updated to a candidate template and its risk score will be increased. If the number of conflicts of a candidate template reaches the preset risk threshold, the template status of the log template will be updated to a high-risk template and its risk score will be increased. If a high-risk template does not form multiple conflict patterns that can be split, and its conflict count increases continuously until the number of consecutive increases reaches a preset obsolescence threshold, then the template status of the log template will be updated to obsolescence and it will be removed from the cache. If the number of conflicts of a high-risk template reaches the preset threshold for splitting, the template status of the high-risk template will be updated to a template to be split, and the logs that conflict with the template to be split and the corresponding log templates will be recorded. If the recorded logs correspond to multiple incompatible log templates, the template to be split is split into multiple new candidate templates, index information of each candidate template is generated and inserted into the template entity index table, the template string and template identifier corresponding to each candidate template are inserted into the template index table, and the original template to be split is removed from the cache. If the recorded log does not yet correspond to multiple incompatible log templates, then the log template remains in a pending split state, and subsequent conflict logs continue to be recorded. Furthermore, in S1, when matching the current token subsequence with the cached log templates, log templates with lower risk scores have higher matching priority.

[0064] Based on the above evolutionary operations, when a log entry is consistently hit by the cache, and the large language model is not invoked and the template remains unchanged, the cache evolution module increases the consistent hit count of the corresponding template, reduces its risk score, and marks it as a trusted template when the consistent hit count reaches a preset threshold. This operation is used to enhance the priority of high-frequency consistent templates.

[0065] When a log entry, after being corrected by the large language model, has a high similarity to a candidate cache template, but differs in some tokens, the cache evolution module determines whether this difference is a generalizable parameter difference. If the difference corresponds to an IP address, number, path, identifier, or other runtime variable, the position is generalized to a parameter placeholder, and the original template is updated. This operation reduces template fragmentation. For example, if the original template is "Task 1 starts on node A" and the new template is "Task 2 starts on node B", the system can generalize the difference to "Task 1 starts on node A". <num>At node< Startup.

[0066] When the final template obtained after a log entry is corrected by the large language model has low similarity to the cache candidate template, or the differences are concentrated in the static token representing the semantics of the log event, the cache evolution module considers that the current candidate template may not be suitable for parsing the log. In this case, the system increases the conflict count of the candidate template, raising its risk score, and marks the template as a risky template when the conflict count reaches a downgrade threshold. Risky templates have lower priority in subsequent cache matching.

[0067] When a risk template continuously conflicts with multiple different final templates, and the number of conflicts reaches a splitting threshold, the cache evolution module marks the template as a template to be split and performs the splitting. The system generates new template records based on the different final templates appearing in the conflict logs, and initializes the template identifier, source, representative log sample, risk score, and update time for the new template. This operation is used to avoid incorrectly merging semantically different log events into the same template.

[0068] When a final template cannot be merged with any existing template, the cache evolution module inserts it as a new template into the cache. When inserting a new template, the system writes the template into the prefix tree path, the template entity index table, and the normalized template index table, and saves representative log samples. If the template comes from the correction results of a large language model, it can be assigned a higher initial confidence score, but its source of invocation is also recorded; if the template comes from a small language model and there is no conflict, it can be assigned a lower risk score.

[0069] In one specific embodiment, the system parses the log "Failed to connect to 10.0.0.1 at port8080". The preprocessing module normalizes it to "Failed to connect to". <ip>at port <num>The template cache matching module first checks if a matching normalized template exists. If the template exists in the cache, it is directly matched, and the template "Failed to connect to" is output. <ip>at port <num>The template is then processed, and the parameters "10.0.0.1" and "8080" are extracted. If the template does not exist in the cache, the small language model generates the initial template. This process is repeated when generating the token "connect" to. <ip>”"at”"port”" <num>During the process, the neural routing module determines whether each token requires the intervention of the large language model. If the small language model is in the process... <ip>"If the position confidence is high, the output of the small language model will be accepted; if a '<' appears in the small model candidate after 'port'..." > and " <num>"If the location is uncertain and the routing probability exceeds the threshold, the large language model is invoked to determine whether a numeric placeholder should be used at that position. After the final template is generated, the cache evolution module inserts the template into the cache so that subsequent logs of the same type can directly hit it."

[0070] In another specific embodiment, the template "User<" already exists in the cache. If the message "login success" is followed by the log "User admin login failed", and the smaller language model tends to parse it as a similar form of an existing template, but the larger language model corrects this by recognizing "success" and "failed" as representing different event semantics, then the cache evolution module will not merge the two, but will instead retain or add the template "User< "Login failed". Since the difference occurs on the static semantic token, this difference should not be generalized to parameters. Through this mechanism, the present invention can avoid log event confusion caused by overgeneralization.

[0071] In yet another specific embodiment, the template "Read block" exists in the cache. <num>from node< The message ">" was followed by multiple log entries: "Read block 100 from node A" and "Read block 101 from node B". If these log entries consistently hit the cache, the system continuously increases the number of consistent hits for that template and decreases its risk score. Once the template becomes a trusted template, it receives higher priority during candidate matching, thereby improving the efficiency of high-frequency template matching.

[0072] It is easy to understand that the parameter size of a small language model is much smaller than that of a large language model. The parameter size of the two differs by 1 to 3 orders of magnitude, or even more. In practical applications, the large language model and the small language model can be selected according to the specific prediction needs and system resources.

[0073] Example 2: A computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the log parsing method for size-model coordinating provided in Embodiment 1.

[0074] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / num> < / num> < / ip> < / num> < / ip> < / num> < / ip> < / num> < / ip> < / num> < / num> < / ip>

Claims

1. A log parsing method based on a combination of large and small models, characterized in that, include: The log text to be parsed is preprocessed into a token sequence, and the positional mapping relationship between each token and the original log text is recorded. For each log entry, extract the corresponding token subsequence and perform the log parsing step; the log parsing step includes: S1: Match the current token subsequence with the cached log template. If the match is successful, use the current log template as the current log template and output the corresponding template string and parameter position. The parsing of the current log ends. Otherwise, select the first token in the current token subsequence as the target token, initialize the template prefix to empty, and go to S2. S2: Use a small language model to parse the template token corresponding to the target token; S3: Use the trained routing model to predict the route probability that needs to be corrected for the template token parsed by the small language model. If the route probability is greater than the preset routing threshold, proceed to S4; otherwise, add a template prefix to the template token output by the small language model and proceed to S5. The routing model is a neural network model; S4: Use a large language model to parse the template token corresponding to the target token and add a template prefix; S5: If the current token subsequence has been processed, proceed to S6; otherwise, select the next token as the target token and proceed to S2. S6: Use the current template prefix as the log template for the current log, and output the corresponding template string and parameter positions. The parsing of the current log ends.

2. The log parsing method for collaborative large and small models as described in claim 1, characterized in that, S1 also includes: If the current token subsequence does not match the cached log template, the similarity between the current token subsequence and each cached log template is calculated as the candidate score of each log template, and the log template with the candidate score higher than the preset candidate score is taken as the fuzzy hit result. Furthermore, in S4, when using the large language model to parse the template token corresponding to the target token, the input of the large language model includes: the original log, the current template prefix, the template token parsed by the small language model, the fuzzy hit result, and the prompt information.

3. The log parsing method for collaborative large and small models as described in claim 2, characterized in that, In S2, the small language model parses the template token corresponding to the target token and outputs the features of the template token. Furthermore, in S3, when using the pre-trained routing model to predict the route probability that needs to be corrected for the template token parsed by the small language model, the input of the routing model includes: the template token parsed by the small language model, the features of the template token, the log template with the highest candidate score in the fuzzy hit results, and the number of log templates in the fuzzy hit results.

4. The log parsing method for collaborative large and small models as described in claim 3, characterized in that, The training methods for the routing model include: Collect historical log text, preprocess it into a token sequence, and then use the small language model and the large language model to parse the template token corresponding to each token, and compare the parsing differences between the small language model and the large language model for each token. If the parsing results of the small language model and the large language model are consistent for the same token, then the corresponding token position is marked as no routing required, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, but the difference does not affect the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as neutral difference, and the routing probability is marked as 0; if the parsing results of the small language model and the large language model for the same token differ, and the difference affects the structure of the log template, parameter boundaries, and log event classification results, then the corresponding token position is marked as critical divergence, and the routing probability is marked as 1. A neural network model is trained using the labeled token sequence to obtain a trained routing model.

5. The log parsing method for collaborative large and small models as described in any one of claims 1 to 4, characterized in that, The log parsing results include: log template, template identifier, template string, parameter position, parsing source, cache hit identifier, routing information, and parsing confidence. Among them, the template identifier is used to uniquely identify the log template; the parsing source is used to mark whether each template token in the log template is parsed by the large language model or the small language model; the cache hit identifier is used to mark whether the log template hits the cache; the routing information is used to record the number of template tokens in the log template parsed by the large language model; and the parsing confidence is the weighted sum of the routing probabilities of each template token.

6. The log parsing method for collaborative large and small models as described in any one of claims 1 to 4, characterized in that, In S6, before using the current template prefix as the log template for the current log and outputting the corresponding parameter position, it also includes: merging consecutive template tokens identified as parameters in the current template prefix into a single parameter placeholder and updating the parameter position information.

7. The log parsing method for collaborative large and small models as described in any one of claims 1 to 4, characterized in that, Log templates are stored in the cache in the form of a prefix tree, and a template entity index table is built in the cache; The template entity index table is used to store the index information of each log template; The index information includes: template string, number of hits, number of conflicts, risk score, template status, and associated logs.

8. The log parsing method for collaborative large and small models as described in claim 7, characterized in that, A template index table is also established in the cache; the template index table is used to record the template identifier and the corresponding template string of each template in the cache; Furthermore, in S1, the current token subsequence is matched against the cached log template, including: S11: Search the template index table according to the string corresponding to the current log. If there is a template string in the template index table that is completely consistent with the string, the match is successful and proceed to S13; otherwise, proceed to S12. S12: Query the current token subsequence in the prefix tree. If the query is successful, the match is considered successful and proceed to S13; otherwise, the match is considered unsuccessful and proceed to S13. S13: Matching ends.

9. The log parsing method for collaborative large and small models as described in claim 8, characterized in that, S6 further includes: performing a template evolution operation based on the parsing result; the template evolution operation includes: If the current parsing result does not hit the cache, perform the following operations: If the log template corresponding to the current log is the same as the cached log template t1, then update the index information of log template t1; If the current log template is inconsistent with the cached log template, but its similarity with the cached log template t2 is higher than the preset similarity threshold, and the difference between the two is caused by the change of parameter value, then the difference position in the log template t2 is generalized to parameter placeholders, and the index information of the generalized log template t2 is updated to realize the merging of log templates. If the log template corresponding to the current log is inconsistent with both the cached log templates and cannot be merged with the cached log templates, then the log template corresponding to the current log is inserted as a new log template into the prefix tree, and the index information of the log template corresponding to the current log is generated and inserted into the template entity index table; in the generated index information, the hit count is 0, the conflict count is 0, the template status is candidate template, and the associated log is the current log. Insert the template string and template identifier corresponding to the log template into the template index table; If the current parsing result hits the cache, then perform the following operations: If a conflict is detected in the hit results, the conflict count of the hit log template is incremented by 1 and its risk score is increased; otherwise, the hit count of the hit log template is incremented by 1 and its risk score is decreased. If the number of hits of a candidate template increases continuously, and the number of consecutive increases reaches a preset stable threshold, then the template status of the log template will be updated to a stable template, and its risk score will be reduced. If the number of hits of a stable template increases continuously, and the number of consecutive increases reaches a preset support threshold, then the template status of the stable template is updated to frozen template, and its risk score is reduced; the status of the frozen template is no longer updated. If the number of conflicts of a stable template increases and the increase reaches a preset backoff threshold, the template status of the stable template will be updated to a candidate template and its risk score will be increased. If the number of conflicts of a candidate template reaches the preset risk threshold, the template status of the log template will be updated to a high-risk template and its risk score will be increased. If a high-risk template does not form multiple conflict patterns that can be split, and its conflict count increases continuously until the number of consecutive increases reaches a preset obsolescence threshold, then the template status of the log template will be updated to obsolescence and it will be removed from the cache. If the number of conflicts of a high-risk template reaches the preset threshold for splitting, the template status of the high-risk template will be updated to a template to be split, and the logs that conflict with the template to be split and the corresponding log templates will be recorded. If the recorded logs correspond to multiple incompatible log templates, the template to be split is split into multiple new candidate templates, index information of each candidate template is generated and inserted into the template entity index table, the template string and template identifier corresponding to each candidate template are inserted into the template index table, and the original template to be split is removed from the cache. If the recorded log does not yet correspond to multiple incompatible log templates, then the log template remains in a pending split state, and subsequent conflict logs continue to be recorded. Furthermore, in S1, when matching the current token subsequence with the cached log templates, log templates with lower risk scores have higher matching priority.

10. A computer-readable storage medium, characterized in that, The system includes a stored computer program, which, when executed by a processor, implements the log parsing method for size-model collaboration as described in any one of claims 1 to 9.

Images