A semantic log analysis system based on a large language model

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a semantic log analysis system based on a large language model. The system comprises a subword feature extraction module, which deletes a cause and effect mask to adapt the structure of LLAMA2 to an analysis task, and extracts accurate feature representations for each subword by using the rich knowledge of the large language model; a semantic recognition module, which integrates the features of each subword based on a multi-head cross attention mechanism, and then extracts a log template and predicts the semantic category of a log parameter; and an analysis tree module, which stores the analyzed template in a tree structure, and constructs and updates the template synchronously with the analysis process, and is used for accelerating the reasoning process of large-scale real-time logs. The application achieves satisfactory results on conventional analysis and semantic analysis datasets. The application has strong generalizability, can capture the semantic features of logs, effectively alleviates the problem of low analysis efficiency under large-scale log data, and can provide strong support for downstream tasks related to logs.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a semantic log parsing system based on a large language model. Background Technology

[0002] Log parsing is a crucial step in automated log analysis, aiming to transform unstructured or semi-structured raw logs into a structured format for further management, understanding, and analysis. Current log parsing methods can be broadly categorized into three types: unsupervised methods, supervised methods, and methods based on large language models. Unsupervised methods use strategies such as frequent pattern mining, log clustering, and heuristics to extract invariant components from logs as templates. Supervised methods utilize natural language processing techniques, training models using labeled log data. Both types of methods suffer from poor generalization: unsupervised methods require rule adjustments and numerous hyperparameter adjustments as logs change, while supervised methods require model retraining. Methods based on large language models leverage pre-trained knowledge from large language models, improving model generalization. However, most existing log parsers based on large language models rely on OpenAI's online API, making them highly susceptible to network latency fluctuations and ChatGPT version changes. Furthermore, log data often contains a large amount of private business information, and using online APIs can lead to privacy concerns.

[0003] On the other hand, traditional log parsers focus on extracting log templates while ignoring the rich semantic information contained in log variables. In other words, they can only distinguish between constants and variables in logs, without further understanding and analysis of the variables. With the development of artificial intelligence in the IT industry, the demand for downstream tasks such as anomaly detection, root cause analysis, and log summarization is increasing, and traditional parsers that only extract log templates can no longer meet these needs.

[0004] Furthermore, with the rapid development of software systems, logs in real-world scenarios are becoming increasingly massive and complex. In real production environments, software systems generate tens of thousands of logs per second. Although existing supervised parsers have surpassed traditional unsupervised parsers in parsing accuracy, their low operating efficiency limits their application in real-world scenarios. Log parsers based on large language models face the same challenge: despite their powerful functionality and good generalization ability, their high invocation costs and runtime overhead make it impractical to directly use large language models to parse large amounts of real-time logs.

[0005] In summary, there is an urgent need to design an offline log parsing method that exhibits good generalization ability, maintains high parsing accuracy and stability across log sources, and requires no additional adjustments. It needs semantic log parsing capabilities, enabling it to predict the semantics of log parameters while extracting log templates, and simultaneously output semantic tags for both log templates and log parameters to downstream tasks. Furthermore, it needs to maintain high parsing efficiency when dealing with large-scale real-time logs from large software systems. Summary of the Invention

[0006] The purpose of this invention is to provide a semantic log parsing system based on a large language model that offers high accuracy and stability.

[0007] The semantic log parsing system based on a large language model provided by this invention includes the following modules: a sub-word feature extraction module, a semantic recognition module, and a parse tree module; wherein:

[0008] The sub-word feature extraction module is used for word segmentation and log sub-word feature extraction; and log parsing is treated as a sequence labeling task; by removing the causal mask, the structure of LLAMA2 is adapted to the parsing task, and the rich knowledge of the large language model is used to extract accurate feature representations for each sub-word;

[0009] The semantic recognition module, based on a multi-head cross-attention mechanism, integrates the features of each sub-word, and then extracts the log template and predicts the semantic category of the log parameters;

[0010] The parsing tree module stores the parsed templates in a tree structure and is built and updated synchronously with the parsing process to accelerate the inference process of large-scale real-time logs.

[0011] Furthermore:

[0012] The sub-word feature extraction module includes an LLAMA2 tokenizer and an LLAMA2 model with causal mask removed (hereinafter referred to as LogLLAMA). LogLLAMA is appended to the LLAMA2 tokenizer. The LLAMA2 tokenizer takes an original log statement as input and divides it into several words, each word into several sub-words, and each sub-word is identified by a unique identifier. The number of sub-words in each log statement is limited; if there are too many, they are truncated; if there are too few, they are filled with specific sub-words. LogLLAMA takes these sub-words as input and extracts the features of each sub-word.

[0013] Large language models, including LLAMA2, are typically designed for text generation tasks. However, directly using these models to generate templates from logs often leads to errors such as incomplete parameter recognition, inclusion of irrelevant text, or misaligned log structures. For log parsing tasks, accuracy is crucial; even minor deviations can severely impact downstream tasks like anomaly detection. Therefore, text generation methods are unsuitable for log parsing; this invention treats log parsing as a sequence labeling task.

[0014] On the other hand, directly applying fine-tuning instructions to sequence labeling tasks in large language models leads to poor results. This is because large language models typically employ a decoder-only autoregressive architecture, using causal masks in the decoder to prevent information leakage during text generation. This mechanism limits the model's access to unidirectional information, restricting its ability to perceive context when extracting sub-word features.

[0015] Therefore, in this invention, LogLLAMA modifies the structure of LLAMA2, as follows: Figure 2 As shown; specifically, LLAMA2 includes N multi-head self-attention layers with causal masks, while LogLLAMA includes N multi-head self-attention layers, i.e., the causal masks are removed; removing the causal masks allows the model to perceive the entire context of the log when extracting features of the current subword. This design ensures that the model has a consistent and comprehensive understanding of the log, improving the performance of large language models in log parsing.

[0016] During the training phase, LogLLAMA uses the LoRa method for efficient parameter fine-tuning to adapt it to log parsing tasks. During the inference phase, it can be directly applied to log data from various sources without requiring additional fine-tuning.

[0017] The semantic recognition module includes a multi-head attention network and a fully connected network, with the fully connected network following the multi-head attention network.

[0018] Conventional sequence labeling tasks use the first subword of each word for feature extraction and classification. However, this method is unsuitable for log parsing because log words are typically much longer than regular words, and using only the first subword leads to semantic loss, hindering subsequent semantic log parsing. To address this issue, the semantic recognition module of this invention employs a multi-head cross-attention mechanism for feature fusion, ensuring that word features incorporate the features of each subword.

[0019] like Figure 1 As shown, the multi-head attention network aggregates all sub-word features of each word into a single word feature. Specifically, for the i-th log word, its corresponding word feature is... The calculation is as follows:

[0020]

[0021] Among them, V SA Additional, learnable aggregation vectors are added to aggregate word features; as training progresses, V SA Its learnability allows it to continuously optimize based on training logs, making it increasingly suitable for semantic parsing. i1 H i2 ,…,H in Features for each subword; W Q W K W V V represents the learnable weight parameters; Attn() is the standard multi-head attention function. When performing feature aggregation for each word, V... SA W Q W K W V The parameters involved in the Attn() function are shared to enhance the model's transfer learning and pattern recognition capabilities. Specifically, the standard multi-head attention calculation formula is:

[0022] Attn(Q,K,V)=Concat(head1,…,head h W O (2)

[0023] The function Concat() represents matrix concatenation, W O The parameters are learnable, head1,…,head h The calculation formula is as follows, depending on the attention head:

[0024]

[0025]

[0026] Where, d k Let k be the dimension.

[0027] Multi-head attention network outputs features for each word The fully connected network performs log semantic classification for each word based on these features. Assuming the total number of semantic categories for a word is K, the fully connected layer maps the word features to K dimensions. In this invention, the total number of semantic categories K is 17, including one template constant category and 16 parameter semantic categories, specifically defined as follows:

[0028]

[0029]

[0030] During training, the cross-entropy loss CE_Loss is calculated for each word. Specifically, for a given word, its word features are: Then its loss function is:

[0031]

[0032] Where K is the total number of semantic categories for words, and FC() is a fully connected network. Predict the probability that the word belongs to label i for the model. i The actual label corresponding to the word, i.e.:

[0033]

[0034] During the inference phase, considering the ambiguity and polysemy of log semantics, the following prediction strategy is adopted: for words predicted as parameters (the category corresponding to the highest prediction probability belongs to the parameter), the top three parameter semantic categories with the highest prediction probabilities are retained as the results; for words predicted as template constants, their template constant categories are retained as the prediction results.

[0035] The parse tree module uses a tree structure similar to but different from the prefix tree to store log templates: the root node of the parse tree does not store content; each intermediate node is a template word or a placeholder “<*>” representing a parameter; each leaf node is a log template library that stores several log templates containing parameter semantic tags.

[0036] This invention introduces a parse tree module to accelerate model inference, based on the fact that in real-world scenarios, many logs are identical, or differ only slightly in a few parameters, because they originate from the same log template. Therefore, a new log entry likely corresponds to a previously parsed template. If each obtained template can be stored during inference and queried before each parsing step, the result can be obtained directly, thus accelerating the inference process.

[0037] like Figure 3 As shown, the parse tree is constructed as follows:

[0038] For templates where the first word is a constant, the first word is used as the first-level intermediate node, and the log template is stored in the leaf node of that intermediate node. For templates where the first word is a parameter, the "<*>" key is used as the first-level intermediate node; if the second word is a constant, the second word is used as the second-level intermediate node, and the log template is stored in the leaf node of the second-level intermediate node; if the second word is a parameter, the "<*>" key is used as the second-level intermediate node, and the log template is stored in the leaf node of the second-level intermediate node.

[0039] The construction and updating of the parse tree are synchronized with the model inference process. When a new log arrives, it first enters the parse tree for log template matching. If the search can proceed along the tree to a leaf node, the similarity between the log and each template in the template library of the current leaf node is calculated. The formula for calculating the similarity between logs and templates is as follows:

[0040]

[0041] Here, the function S() represents word segmentation of the log or template by spaces, and the function len() calculates the number of words after word segmentation. Three (if any) most similar log templates are selected and matched against the input log using regular expressions. If a match is found, the semantic tags of the template and its parameters are directly output as the prediction result. If no template matches successfully, or the search cannot reach a leaf node, the log is sent to the parser for parsing, and the parse tree is updated based on the prediction result, storing the new log template and its corresponding semantic tags in the parse tree.

[0042] This invention has at least the following beneficial effects:

[0043] This invention designs a semantic log parsing method based on a large language model. An improved LLAMA2 structure is used to apply the rich pre-trained knowledge of the large model to the log parsing method, enhancing the model's generalization ability. The semantic parsing module enables the model to predict the semantic labels of parameters while extracting log templates, adapting to the needs of increasingly complex downstream tasks. The parse tree module accelerates the large-scale real-time log inference process. Unlike most log parsers based on large language models, this invention can run offline, avoiding problems such as network latency, interface changes, and data security. After fine-tuning using log data from several different sources, this invention can directly parse logs from other sources without additional training.

[0044] Other advantages, objectives and features of the present invention will be apparent in part from the following description, and in part from the understanding of those skilled in the art through study and practice of the invention. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the semantic log parsing system based on a large language model according to the present invention.

[0046] Figure 2 This is a structural comparison diagram of LLAMA2 and LogLLAMA.

[0047] Figure 3 This is a diagram of the parsing tree module structure. Detailed Implementation

[0048] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0049] This invention provides a semantic log parsing method based on a large language model, which includes the following modules:

[0050] (1) The sub-word feature extraction module includes an LLAMA2 tokenizer and an LLAMA2 model with causal mask removed (hereinafter referred to as LogLLAMA). LogLLAMA is connected after the LLAMA2 tokenizer. The LLAMA2 tokenizer takes an original log statement as input and divides it into several words, each of which is further divided into several sub-words. Subsequently, LogLLAMA extracts the features of these sub-words. By removing the causal mask, LogLLAMA modifies the structure of LLAMA2, enabling it to perceive the context of each word and adapt to the log parsing task.

[0051] (2) A semantic recognition module, comprising a multi-head attention network and a fully connected network, with the fully connected network following the multi-head attention network. The multi-head attention network aggregates all subwords of each word into a single word feature, and the fully connected network performs semantic classification of the word based on these features. Further, this invention designs 17 categories, including one template constant category and 16 parametric semantic categories. Cross-entropy loss is used during the training phase. During the inference phase, if the category corresponding to the word's highest predicted probability is the template constant category, it is predicted as the template constant; if the category corresponding to the word's highest predicted probability is the parametric semantic category, it is predicted as the parametric category, and the three semantic categories with the highest predicted probabilities are taken as the prediction results.

[0052] (3) The parse tree module uses a tree structure similar to a prefix tree to store the parsed log templates. Its construction and updates are synchronized with the log inference process. Before parsing, the three most similar log templates are selected from all templates in the parse tree for regular expression matching. If the match is successful, the template is directly output as the prediction result; if no template can be successfully matched, the log is sent to the parser for parsing, and the parsed template is stored in the parse tree.

[0053] The specific model training and inference steps are as follows:

[0054] (1) Parsing tree query

[0055] When a new log arrives, it is first entered into the parse tree for log template matching. If the search can proceed along the tree to a leaf node, the log is compared with each template in the template library of the current leaf node for similarity. The formula for calculating the similarity between logs and templates is as follows:

[0056]

[0057] Here, the function S() represents word segmentation of the log or template by spaces, and the function len() calculates the number of words after word segmentation of the log or template by spaces. Three (if any) most similar log templates are selected and matched against the input log using regular expressions. If a match is found, the semantic tags of the template and its parameters are directly output as the prediction result. If no template matches successfully, or the search cannot reach the leaf nodes, the log is sent to the parser for parsing.

[0058] (2) Sub-word feature extraction

[0059] The sub-word feature extraction module includes an LLAMA2 tokenizer and an LLAMA2 model with causal mask removed (hereinafter referred to as LogLLAMA). LogLLAMA follows the LLAMA2 tokenizer. The LLAMA2 tokenizer takes an original log statement as input and divides it into several words, each word into several sub-words, and each sub-word is identified by a unique identifier. The number of sub-words in each log statement is limited; if there are too many, they are truncated; if there are too few, they are padded with specific sub-words.

[0060] LogLLAMA accepts these subwords as input and extracts features from each subword. For example... Figure 2 As shown, LogLLAMA modifies the structure of LLAMA2 by removing the causal mask, enabling the model to be aware of the entire log context when extracting features from the current subword. This design ensures a consistent and comprehensive understanding of the log, improving the performance of large language models in log parsing.

[0061] During the training phase, LogLLAMA uses the LoRa method for efficient parameter fine-tuning to adapt it to log parsing tasks. During the inference phase, it can be directly applied to log data from various sources without requiring additional fine-tuning.

[0062] (3) Semantic recognition

[0063] The semantic recognition module includes a multi-head attention network and a fully connected network, with the fully connected network following the multi-head attention network.

[0064] like Figure 1 As shown, the multi-head attention network aggregates all sub-word features of each word into a single word feature. Specifically, for the i-th log word, its corresponding word feature is... The calculation is as follows:

[0065]

[0066] Among them, V SA Additional, learnable aggregation vectors are added to aggregate word features; as training progresses, V SAIts learnability allows it to continuously optimize based on training logs, making it increasingly suitable for semantic parsing. i1 H i2 ,…,H in Features for each subword; W Q W K W V The weights are learnable parameters; Attn() is the standard multi-head attention function. When performing feature aggregation for each word, V... SA W Q W K W V The parameters involved in the Attn() function are shared to enhance the model's transfer learning and pattern recognition capabilities. Specifically, the standard multi-head attention calculation formula is...

[0067] Attn(Q,K,V)=Concat(head1,…,head h W O

[0068] The function Concat() represents matrix concatenation, W O The parameters are learnable, head1,…,head h The calculation formula is as follows, depending on the attention head:

[0069]

[0070]

[0071] Where, d k Let k be the dimension.

[0072] Multi-head attention network outputs features for each word The fully connected network performs log semantic classification for each word based on these features. Assuming the total number of semantic categories for a word is K, the fully connected layer maps the word features to K dimensions. In this method, K = 17, containing one template constant category and 16 parameter semantic categories, specifically defined as follows:

[0073] Serial number Category Semantic category 0 O Template constant 1 OID Unique identifier 2 OBN Name 3 LOI Address 4 TID Type 5 SID Status 6 TPA Time point 7 TDA Time range 8 OBA Quantity 9 OBO Order 10 CRS Resource number 11 OBD Data content 12 CMD Instruction 13 INR Interval 14 VER Version information 15 ERR Error or exception information 16 OTP Others .

[0074] During training, the cross-entropy loss CE_Loss is calculated for each word. Specifically, for a given word, its word features are: Then its loss function is:

[0075]

[0076] Where K = 17 represents the total number of semantic categories for the words, and the function FC() is a fully connected network. Predict the probability that the word belongs to label i for the model. i The actual label corresponding to the word, i.e.:

[0077]

[0078] During the inference phase, due to the ambiguity and polysemy of log semantics, the following prediction strategy will be adopted: for words predicted as parameters (i.e., the category corresponding to the highest prediction probability belongs to the parameter), the top three parameter semantic categories with the highest prediction probabilities will be retained as the result; for words predicted as template constants, their template constant categories will be retained as the prediction result.

[0079] (4) Parsing tree update

[0080] After parsing is complete, the parse tree is updated according to the storage rules of the parse tree, and the predicted template and its parameter semantic labels are stored in the template library of a leaf node of the parse tree.

[0081] Test and verification of the present invention

[0082] This invention selects two datasets, LogPub and LogBase, to test the performance of this application on a regular log parsing dataset and a semantic log parsing dataset.

[0083] LogPub, released by the LogPAI team, is a well-known dataset for general log parsing, containing 50.4 million logs from 14 sources and approximately 3,500 log templates. LogPub only distinguishes between templates and parameters, without further differentiating the semantic categories of the parameters. Therefore, we added semantic annotations to its data.

[0084] LogBase is a dataset we built specifically for semantic log parsing tasks. The construction steps are as follows:

[0085] (1) Use a web crawler to crawl the 121 projects with the most stars from GitHub.

[0086] (2) Use JavaParser to build a log template mining algorithm to extract all possible log templates from the project source code.

[0087] (3) Design tips: Use contextual learning to enable large language models to generate simulated logs based on these templates.

[0088] (4) Two experienced software engineers and four graduate students were invited to screen, label, and cross-validate these simulated logs. LogBase contains 1.1 million logs from 121 sources and approximately 74,500 log templates.

[0089] To verify the superiority of this method, this embodiment is compared with the following existing log parsing methods on a public dataset:

[0090] Drain (from "He P, Zhu J, Zheng Z, et al. Drain: An online log parsing approach with fixed depth tree [C] / / 2017IEEE international conference on webservices(ICWS).IEEE, 2017:33-40.");

[0091] UniParser (excerpted from "Liu Y, Zhang

[0092] LogPPT (from "Le VH, Zhang H. Log parsing with prompt-based few-shotlearning[C] / / 2023IEEE / ACM 45th International Conference on SoftwareEngineering(ICSE).IEEE,2023:2438-2449."); LILAC (from "Jiang Z, Liu J, Chen Z, etal. Llmparser: A llm-based log parsing framework[J].arXiv preprint arXiv:2310.01796,2023.");

[0093] LLMParser (from "Ma Z, Chen AR, Kim DJ, et al. LLMParser: An ExploratoryStudyon Using Large Language Models for Log Parsing[C] / / 2024IEEE / ACM 46thInternationalConference on Software Engineering(ICSE).IEEE Computer Society, 2024:883-883.");

[0094] LILAC (excerpted from "Jiang Z, Liu J, Chen Z, et al. Llmparser:A llm-based logparsingframework[J].arXiv preprint arXiv:2310.01796,2023.");

[0095] VALB (Excerpted from "Li Z, Luo C, Chen TH, et al. Did we miss something important? studying and exploring variable-aware log abstraction[C] / / 2023IEEE / ACM 45thInternationalConference on Software Engineering (ICSE). IEEE, 2023:830-842.").

[0096] This embodiment uses GA, PA, and FTA as metrics for regular log parsing, and SPA and SPA+ as metrics for semantic log parsing. SPA represents the proportion of correctly parsed logs out of the total number of logs. A log is considered correctly parsed if the model correctly distinguishes between the log template and the parameters, and for each parameter, the model's semantic prediction result includes the true semantic category of the parameter. SPA+ is similar to SPA, but more stringent; it is considered correctly parsed only when the label with the highest output probability in the model matches the true semantic category. The results on LogPub and LogBase are shown in Table 1 and Table 2, respectively.

[0097] On LogPub, this invention achieved the highest average performance across all metrics. In terms of grouping performance, the average GA of this invention was 93.3%, 0.6% higher than LILAC. Regarding conventional parsing performance, the PA and FTA of this invention were 93.6% and 84.4%, respectively, exceeding LILAC by 11.5% and 0.2%, and exceeding LogPPT by 27.2% and 76.6%. This indicates that this invention can effectively extract log templates and log parameters. In semantic parsing, both SPA and SPA+ of this invention significantly outperformed the current best semantic log parser, VALB: SPA at 83.2% and SPA+ at 55.9%, exceeding VALB by 29% and 22.1%, respectively. These results demonstrate that this invention can accurately extract log templates while precisely identifying the meaning of log parameters.

[0098] On LogBase, this invention also achieves the highest average performance across all metrics. In terms of grouping performance, SemanticLog achieves an average GA of 75.6%, 8.2% higher than VALB. For traditional log parsing, its PA and FTA are 82.4% and 56.4%, respectively, exceeding VALB by 9.9% and 30%. In semantic log parsing, the invention's SPA and SPA+ are 77.4% and 66%, respectively, surpassing VALB by 15.4% and 31%. Given that LogBase is significantly more complex than LogPub, these results demonstrate that this invention possesses excellent generalization capabilities and can accurately handle large-scale, complex log data.

[0099] Table 1 Performance Comparison on LogPub

[0100]

[0101] Table 2 Performance Comparison on LogBase

[0102]

Claims

1. A semantic log parsing system based on a large language model, characterized in that, It includes a sub-word feature extraction module, a semantic recognition module, and a parse tree module; among which: The sub-word feature extraction module is used for word segmentation and log sub-word feature extraction; and log parsing is treated as a sequence labeling task; by removing the causal mask, the structure of LLAMA2 is adapted to the parsing task, and the rich knowledge of the large language model is used to extract accurate feature representations for each sub-word; The semantic recognition module, based on a multi-head cross-attention mechanism, integrates the features of each sub-word, and then extracts the log template and predicts the semantic category of the log parameters; The parsing tree module stores the parsed templates in a tree structure, and is built and updated synchronously with the parsing process to accelerate the inference process of large-scale real-time logs. The sub-word feature extraction module includes an LLAMA2 tokenizer and an LLAMA2 model with causal mask removed, denoted as LogLLAMA. LogLLAMA is connected after the LLAMA2 tokenizer. The LLAMA2 tokenizer takes an original log statement as input and divides it into several words, each word into several sub-words, and each sub-word is identified by a unique identifier. The number of sub-words in each log statement is limited; if there are too many, they are truncated; if there are too few, they are filled with specific sub-words. LogLLAMA takes these sub-words as input and extracts the features of each sub-word. Here, the LLAMA2 model includes: N layers of multi-head self-attention layers with causal masks. LogLLAMA is an LLAMA2 model with the causal masks removed, that is, LogLLAMA includes N layers of multi-head self-attention layers; removing the causal masks enables the model to be aware of the entire context of the log when extracting the features of the current sub-word. During the training phase, LogLLAMA uses the Lora method for efficient parameter fine-tuning to adapt it to log parsing tasks; during the inference phase, it is directly applied to log data from various sources without requiring additional fine-tuning. The semantic recognition module includes a multi-head cross-attention network and a fully connected network, with the fully connected network following the multi-head attention network; Multi-head attention networks aggregate all sub-word features of each word into a single word feature; specifically, for the i-th log word, its corresponding word feature... The calculation is as follows: ， （1） in, Additional, learnable aggregation vectors are added to aggregate word features; as training progresses, Its learnability allows it to continuously optimize based on training logs, making it increasingly suitable for semantic parsing; Features for each subword; These are learnable weight parameters; Attn() is the standard multi-head attention function; when performing feature aggregation for each word, The parameters involved in the Attn() function are shared to enhance the model's transfer learning and pattern recognition capabilities; the standard multi-head attention calculation formula is: ， （2） Among them, the function Indicates matrix concatenation. For learnable parameters, The calculation formula is as follows: ， （3） ， （4） in, The dimension is k; Multi-head attention network outputs features for each word The fully connected network performs log semantic classification for each word based on these features; assuming the total number of semantic categories for a word is K, the fully connected layer maps the word features to K dimensions. During training, the cross-entropy loss for each word is calculated. Specifically, for a given word, its word characteristics are: Then its loss function is: ， （5） Where K is the total number of semantic categories for words, and FC() is a fully connected network. Predict the probability that the word belongs to label i for the model; The actual label corresponding to the word, i.e.: ， （6） During the inference phase, considering the ambiguity and polysemy of log semantics, the following prediction strategy is adopted: for words predicted as parameters, i.e., the category corresponding to the highest predicted probability belongs to the parameter, the top three parameter semantic categories with the highest predicted probabilities are retained as the result; for words predicted as template constants, their template constant categories are retained as the prediction result. The parse tree module uses a tree structure similar to a prefix tree to store log templates, but with the following differences: the root node of the parse tree does not store content; each intermediate node is a template word or a placeholder "<*>" representing a parameter; each leaf node is a log template library, storing several log templates containing parameter semantic tags; its construction method is as follows: For templates where the first word is a constant, the first word is used as the first-level intermediate node, and the log template is stored in the leaf node of this intermediate node; for templates where the first word is a parameter, the "<*>" key is used as the first-level intermediate node; if the second word is a constant, the second word is used as the second-level intermediate node, and the log template is stored in the leaf node of the second-level intermediate node; if the second word is a parameter, the "<*>" key is used as the second-level intermediate node, and the log template is stored in the leaf node of the second-level intermediate node. The construction and updating of the parse tree are synchronized with the model inference process; when a new log arrives, it first enters the parse tree for log template matching; if the search can proceed along the tree to a leaf node, the similarity between the log and each template in the template library of the current leaf node is calculated; the similarity calculation formula between logs and templates is as follows: ， （7） The function S() represents segmenting the log or template by spaces, and the function len() calculates the number of words after segmenting the log or template by spaces. The three most similar log templates are selected and matched with the input log using regular expressions. If a match is successful, the semantic tags of the template and its parameters are directly output as the prediction result. If no template is matched, or the search cannot reach the leaf node, the log is sent to the parser for log parsing, and the parse tree is updated according to the prediction result. The new log template and its corresponding parameter semantic tags are stored in the parse tree.

2. The semantic log parsing system based on a large language model according to claim 1, characterized in that, The total number of semantic categories K is 17, including one template constant category and 16 parameter semantic categories, as defined below: 。

Images