Log sequence anomaly detection method based on enhanced knowledge base and dynamic mask modeling

Abstract

The invention discloses a log sequence anomaly detection method based on an enhanced knowledge base and dynamic mask modeling, and belongs to the technical field of computers. According to the log sequence anomaly detection method and device, the problems that existing log sequence anomaly detection is poor in accuracy and low in efficiency are solved, a pre-trained log anomaly detection model is adopted for carrying out anomaly detection and anomaly root tracking on a to-be-detected log, and the training process of the log anomaly detection model is improved; performing analysis, structuring and content enhancement on the historical logs by adopting a dynamic prompt template, and then generating an enhanced knowledge base; extracting normal and abnormal log sequence samples from the enhanced knowledge base, and dividing the samples into high-similarity sample pairs and low-similarity sample pairs by adopting a cosine similarity algorithm; generating lexical elements from the samples by using a byte pair coding word segmentation device; performing lexical element masking on the high-similarity sample pair and the low-similarity sample pair by adopting a difference sensitive mask and a threshold-constrained random mask; and training the BERT model by adopting the masked lexical elements to obtain a log anomaly detection model. The method is suitable for log sequence anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology. Background Art

[0002] With the rapid development of information technology, the amount of log data generated by various systems and application programs has increased exponentially. These log data not only record the running state information of the system, but may also contain potential fault or abnormal information. Therefore, how to efficiently and accurately identify anomalies from massive logs and locate their roots has become an important research topic. Effective log anomaly detection can significantly improve the stability and security of the system and reduce the losses caused by system failures.

[0003] However, traditional log anomaly detection methods have many limitations when dealing with large-scale, diverse and complex log data. The rule matching method requires a large number of rules to be predefined, which not only makes it difficult to cover all possible abnormal situations, but also has a high cost of maintaining these rules and is prone to false alarms or missed alarms. Although the statistical analysis method can process a large amount of data, it is easily interfered by noise, resulting in frequent false alarms or missed alarms. At the same time, its ability to identify complex patterns is limited, and it cannot effectively distinguish the subtle differences between normal and abnormal logs. In recent years, some machine learning-based methods such as Support Vector Machine (SVM) and Isolation Forest have been applied to log anomaly detection, but they perform poorly when facing high-dimensional sparse data and have a poor processing effect on unbalanced data sets. For example, in the HDFS data set, due to the very simple log structure, using a pre-trained language model may instead lead to a performance decline, which indicates that directly applying existing models cannot completely solve the problem of log anomaly detection.

[0004] In recent years, pre-trained language models based on the Transformer architecture (such as BERT) have achieved remarkable success in natural language processing tasks. BERT has achieved a comprehensive understanding of the context through a bidirectional encoder and has performed well in a variety of NLP tasks. Nevertheless, directly applying BERT to log anomaly detection still faces challenges. First, existing methods usually lack data augmentation processing, resulting in poor performance of the model on unbalanced data sets. For example, in some application scenarios, the number of abnormal samples is much less than that of normal samples, which will cause the model to be biased towards normal samples, thus affecting the accuracy of anomaly detection. Second, the traditional MLM (Masked Language Modeling) strategy usually randomly selects 15% of the tokens for masking, and this fixed ratio masking strategy cannot adapt to the characteristics of different types of logs. The keyword fields (such as memory capacity, timestamp, etc.) in some log sequences are crucial for anomaly detection, but the probability of these fields being randomly masked is relatively low, thus affecting the learning effect of the model. Summary of the Invention

[0005] The present invention is to solve the problems of poor accuracy and low efficiency in existing log sequence anomaly detection, and now provides a log sequence anomaly detection method based on an enhanced knowledge base and dynamic mask modeling.

[0006] The log sequence anomaly detection method based on an enhanced knowledge base and dynamic mask modeling according to the present invention includes:

[0007] Using a pre-trained log anomaly detection model to perform anomaly detection and anomaly root cause tracking on the log to be detected. The training process of the log anomaly detection model is as follows:

[0008] Step 1: Obtain historical log samples, mark the normality and anomaly of the samples, use the marked historical log samples to construct roles, tasks, and workflows with a dynamic prompt template, and use a pre-trained language model to parse and enhance the content of each marked historical log sample based on the roles, tasks, and workflows, and convert each sample into a positive and negative anomaly sample pair to generate an enhanced knowledge base;

[0009] Step 2: Extract normal log sequence samples and abnormal log sequence samples from the enhanced knowledge base, and use the attention mechanism, encoding, and decoding architecture of Transformer to perform deep encoding and semantic understanding on the log sequence samples to obtain the relevance and deep semantic features between the extracted sample fields;

[0010] Step 3: Use the relevance and deep semantic features between the sample fields, and use the cosine similarity algorithm to calculate the similarity between every two samples in the enhanced knowledge base, obtain the similarity pairs between the samples, and divide the similarity pairs into high-similarity sample pairs and low-similarity sample pairs;

[0011] Step 4: Use the Byte Pair Encoding (BPE) tokenizer to preprocess the high-similarity sample pairs and low-similarity sample pairs to generate high-similarity sample pair tokens and low-similarity sample pair tokens;

[0012] Step 5: Mask the high-similarity sample pair tokens with a difference-sensitive mask, and mask the low-similarity sample pair tokens with a threshold-constrained random mask; obtain the masked normal tokens and abnormal tokens;

[0013] Step 6: Establish a BERT (Bidirectional Encoder Representations from Transformers) model, and use the masked normal tokens and abnormal tokens to train the BERT model to obtain a log anomaly detection model.

[0014] Further, in the present invention, in step one, the dynamic prompt template constructs roles, tasks, and workflows using the marked historical log samples, inputs the roles, tasks, and workflows into a pre-trained language model to perform semantic depth parsing based on questions on the historical log samples, obtains a three-dimensional structured parsing result including an event essence layer, a system impact layer, and a structure type layer, and then uses the parsing result to generate a sample pair including positive and abnormal samples for each sample.

[0015] Further, in the present invention, in step one, the enhanced knowledge base includes log samples marked with normal and abnormal, parsing results, content enhancement results, unique identifiers (ids) of the logs, the essence of the events in the logs, the impact of the events on the system, the structure of the events, and event types;

[0016] The content enhancement results include identified abnormal points of the log samples, abnormal descriptions, abnormal scenarios, and abnormal results.

[0017] Further, in the present invention, the formula for the dynamic prompt template is:

[0018] Prompt = Role + Task + WorkFlow + x log,label

[0019] where Role represents the role of a large model expert, Task represents the set of tasks that the pre-trained language model needs to perform, WorkFlow represents the questions to assist the pre-trained language model in thinking, and x log,label represents the sample log information and its positive and abnormal markings.

[0020] Further, in the present invention, in step three, the method of dividing the samples into high-similarity sample pairs and low-similarity sample pairs is:

[0021]

[0022] where · represents the dot product of vectors, and ∥·∥ represents the norm of the vector; a threshold θ is set, and the samples are divided into high-similarity sample pairs S high and low-similarity sample pairs S low :

[0023] S high = {(s i , s j ) | sim(s i , s j ) > θ}

[0024] S low = {(s i , s j ) | sim(s i , s j ) ≤ θ}

[0025] wherein, s i and s j represent the i-th sample and the j-th sample respectively.

[0026] Furthermore, in the present invention, in step four, the formula of the byte pair encoding tokenizer is:

[0027] s′ = BPE(s)

[0028] wherein, BPE(·) is the byte pair encoding tokenizer, s represents the sample, and s′ represents the token sequence after tokenization.

[0029] Furthermore, in the present invention, it further includes the step of respectively marking normal tokens and abnormal tokens for the tokens after tokenization. In this step, normal sample tokens are marked as s normal , and abnormal sample tokens are marked as s anomaly .

[0030] Furthermore, in the present invention, in step five, the formula for masking high-similarity sample tokens with a difference-sensitive mask is:

[0031]

[0032] wherein, diff(s i , s j ) is a function for identifying the different parts between two log sequences; represents the token after masking of the high-similarity sample mask;

[0033]

[0034] wherein, x is a token in the log sequence.

[0035] Furthermore, in the present invention, in step five, the formula for masking low-similarity sample tokens with a threshold-constrained random mask is:

[0036]

[0037] wherein, random_mask(·) represents the random mask function, and δ is the set threshold range.

[0038] Furthermore, in the present invention, in step six, the BERT model formula is:

[0039] M = train(M, s masked )

[0040] wherein, M represents the output of the BERT model, train( ) represents the model training operation, s masked represents the masked tokens, including and

[0041] The present invention can not only effectively train a BERT model dedicated to log sequence anomaly detection, but also use this model for accurate log anomaly detection and anomaly root cause tracking. This method can not only determine whether a log sequence is abnormal, but also locate the specific part that causes the anomaly, significantly improving the stability and security of the system. And apply the Byte-Pair Encoding (BPE) tokenizer to generate tokens; distinguish normal sample tokens and abnormal sample tokens during the tokenization process, and adopt different processing strategies for different types of samples. For highly similar samples, identify and mask and replace the parts that differ between the abnormal log sequence and the normal log sequence; for low-similarity samples, randomly select logs within a set threshold range for masking and replacement, increasing the data diversity of model training. Effectively improves the accuracy of model anomaly detection. Brief Description of the Drawings

[0042] Figure 1 It is a flowchart of the method described in the present invention. Detailed Embodiments

[0043] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0044] Detailed Embodiment 1: Refer to Figure 1 Specifically describe this embodiment. The method for log sequence anomaly detection based on an enhanced knowledge base and dynamic masking modeling described in this embodiment includes:

[0045] Use a pre-trained log anomaly detection model to perform anomaly detection and anomaly root cause tracking on the log to be detected. The training process of the log anomaly detection model is as follows:

[0046] Use a pre-trained log anomaly detection model to perform anomaly detection and anomaly root cause tracking on the log to be detected. The training process of the log anomaly detection model is as follows:

[0047] Step 1: Obtain historical log samples, mark the normality and abnormality of the samples, use the marked historical log samples to construct roles, tasks, and workflows with a dynamic prompt template, and use a pre-trained language model to parse and strengthen the content of each marked historical log sample based on the roles, tasks, and workflows, and convert each sample into a positive and abnormal sample pair to generate an enhanced knowledge base;

[0048] Step 2: Extract normal log sequence samples and abnormal log sequence samples from the enhanced knowledge base. Use the attention mechanism, encoding, and decoding architecture of Transformer to deeply encode and semantically understand the log sequence samples, and obtain the relevance between the extracted sample fields and the deep semantic features.

[0049] Step 3: Utilize the relevance between the sample fields and the deep semantic features, and adopt the cosine similarity algorithm to calculate the similarity between every two samples in the enhanced knowledge base, obtain the similarity pairs of the samples, and divide the similarity pairs into high-similarity sample pairs and low-similarity sample pairs.

[0050] Step 4: Use the Byte Pair Encoding (BPE) tokenizer to preprocess the high-similarity sample pairs and low-similarity sample pairs to generate high-similarity sample pair tokens and low-similarity sample pair tokens.

[0051] Step 5: Mask the high-similarity sample pair tokens with a difference-sensitive mask, and mask the low-similarity sample pair tokens with a threshold-constrained random mask; obtain the masked normal tokens and abnormal tokens.

[0052] Step 6: Establish a BERT (Bidirectional Encoder Representations from Transformers) model, and use the masked normal tokens and abnormal tokens to train the BERT model to obtain a log anomaly detection model.

[0053] Further, in the present invention, in Step 1, the dynamic prompt template constructs roles, tasks, and workflows using the labeled historical log samples, inputs the roles, tasks, and workflows into the pre-trained language model to perform semantic depth parsing based on questions on the historical log samples, obtains a three-dimensional structured parsing result including an event essence layer, a system impact layer, and a structure type layer, and then uses the parsing result to generate sample pairs including positive and abnormal samples for each sample.

[0054] Further, in the present invention, in Step 1, the enhanced knowledge base includes log samples marked with normal and abnormal, parsing results, content enhancement results, unique identifiers (ids) of the logs, the event essence of the logs, the impact of the event on the system, the structure of the event, and the event type.

[0055] The content enhancement results include identified abnormal points of the log samples, abnormal descriptions, abnormal scenarios, and abnormal results.

[0056] Further, in the present invention, the formula of the dynamic prompt template is:

[0057] Prompt = Role + Task + WorkFlow + x log,label

[0058] Among them, Role represents the role of the large model expert, Task represents the set of tasks that the pre-trained language model needs to execute, WorkFlow represents the questions to assist the pre-trained language model in thinking, and x log,label represents the sample log information and its positive / abnormal label.

[0059] Furthermore, in the present invention, in step three, the method for dividing the samples into high-similar sample pairs and low-similar sample pairs is as follows:

[0060]

[0061] where · represents the dot product of vectors, ∥·∥ represents the norm of vectors; a threshold θ is set, and the samples are divided into high-similar sample pairs S high and low-similar sample pairs S low :

[0062] S high ={(s i , s j ) | sim(s i , s j ) > θ}

[0063] S low ={(s i , s j ) | sim(s i , s j ) ≤ θ}

[0064] where s i and s j respectively represent the i-th sample and the j-th sample.

[0065] Furthermore, in the present invention, in step four, the formula of the byte pair encoding tokenizer is:

[0066] s′ = BPE(s)

[0067] where BPE(·) is the byte pair encoding tokenizer, s represents the sample, and s′ represents the sequence of tokenized tokens.

[0068] Furthermore, in the present invention, it also includes the step of separately labeling the tokenized tokens as normal tokens and abnormal tokens. In this step, the normal sample tokens are labeled as s normal , and the abnormal sample tokens are labeled as s anomaly .

[0069] Furthermore, in the present invention, in step five, the formula for masking the high-similar sample tokens with a difference-sensitive mask:

[0070]

[0071] Among them, diff(s i , s j ) is a function used to identify the different parts between two log sequences; represents the token after the high similarity sample mask;

[0072]

[0073] Among them, x is the token in the log sequence.

[0074] Furthermore, in the present invention, in step five, the formula for masking the low similarity sample tokens using threshold-constrained random masking is:

[0075]

[0076] Among them, random_mask(·) represents the random masking function, and δ is the set threshold range.

[0077] Furthermore, in the present invention, in step six, the BERT model formula is:

[0078] M = train(M, s masked )

[0079] Among them, M represents the BERT model output, train( ) represents the model training operation, s masked represents the masked tokens, including and

[0080] Reference Figure 1 and the detailed description of this embodiment, the log sequence anomaly detection method based on enhanced knowledge base and dynamic masking modeling specifically includes the following steps:

[0081] S1. Structured enhanced knowledge base construction:

[0082] Use predefined dynamic prompt templates to construct roles, tasks, and workflows, and perform in-depth semantic parsing on the original logs based on pre-trained language models (DeepSeek, GPT, Qwen models) to generate a structured enhanced knowledge base, including three-dimensional structured parsing results of the event essence layer, system impact layer, and structure type layer; based on the log information and log knowledge synthesized abnormal log data to balance the proportion of positive and negative samples.

[0083] S2. Similarity feature classification:

[0084] Using the normal and abnormal log sequence samples in the enhanced knowledge base, adopting the attention mechanism, encoding, and decoding architecture of Transformer, deeply encoding and semantically understanding the log sequence sample pairs, capturing the associations between fields, and extracting deep semantic features; adopting the cosine similarity algorithm, and classifying the samples into high-similarity sample pairs and low-similarity sample pairs.

[0085] S3. Use Dynamic Masked Language Modeling

[0086] Implement an improved dynamic masked language modeling. Use the Byte-Pair Encoding (BPE) tokenizer to preprocess the samples to generate tokens, while distinguishing the tokens corresponding to normal samples and abnormal samples; innovatively design a dual-channel masking mechanism; among them, the high-similarity sample pairs perform difference-sensitive masking, and locate and mask the difference feature regions between abnormal logs and normal logs through contrastive learning; the low-similarity sample pairs implement randomly masked with threshold constraints, and perform dynamic masking within a preset similarity range.

[0087] S4. BERT Model for Special Log Sequence Anomaly Detection

[0088] Train a BERT model specifically for log sequence anomaly detection based on the tokens obtained from the above dynamic masking process; use the trained dedicated BERT model to perform log anomaly detection and anomaly root cause tracking, not only judge whether the log sequence is abnormal, but also locate the specific part that causes the anomaly.

[0089] Furthermore, the following features are included in the step S1: using a dynamic prompt template for log parsing and structuring, and log content enhancement, processing the log information; the log content enhancement process uses a large model to generate abnormal log data from normal log data to supplement negative sample pairs and balance the positive and negative sample ratios in the dataset; the finally generated knowledge base contains the parsed and enhanced log information for subsequent anomaly detection and analysis.

[0090] The dynamic prompt template consists of Role, Task, WorkFlow, and the log information to be processed and its label (x log,label )). Specifically, the dynamic prompt template can be expressed as: Prompt = Role + Task + WorkFlow + x log,label , where Role represents the role of the large model expert, Task represents the set of tasks that the large model needs to perform, WorkFlow represents a series of questions to assist the large model in thinking, and x log,label represents the log information to be processed and its label.

[0091] In the log parsing and structuring phase, a series of questions are used to guide the large model (DeepSeek-R1) to understand the key elements of the log. For example, questions such as "What is the essence of the event?", "How will this event affect the system?" help the large model identify the basic information in the log. These parsing results not only provide the basic information of the log but also lay the foundation for subsequent content enhancement. For each log entry L, the parsing result P(L) is generated by the large model, where the parsing result includes the essence of the event, its impact on the system, the structure of the event, and the type of event result. The parsing process can be expressed by the formula: P(L) = f parse (L) where f parse (·) is a function used to generate the parsing result.

[0092] In the log content enhancement phase. Our innovation lies in using normal log data to generate abnormal log data and using abnormal samples to generate normal log data, thereby strengthening negative sample pairs and balancing the ratio of positive and negative samples in the dataset. By asking questions such as "In a given log event, which parts are most likely to be abnormal?", "If this operation fails, what will the log record?", the large model can identify potential abnormal points and generate specific abnormal descriptions. In addition, we also simulate abnormal scenarios to generate detailed abnormal log entries, including timestamps, error codes, stack traces, etc., and describe the specific impacts caused by the anomalies, such as service interruption or performance degradation. This process can be represented by the function f(·): L enhanced = f(L normal ) where L normal is the normal log dataset, and f(·) is a function used to generate abnormal log data L enhanced , thereby strengthening negative sample pairs and balancing the ratio of positive and negative samples in the dataset.

[0093] Finally, the parsed and enhanced log information is integrated into a structured knowledge base. This knowledge base not only contains the original log information but also includes parsing results and enhanced content. Specifically, each log entry has a unique identifier (id), the original log information (log), a label (target) indicating whether it is an abnormal log, the source dataset where this log is recorded, and detailed information on parsing results and enhanced content. The parsing results include the essence of the event, its impact on the system, the structure of the event, and the event type; the enhanced content includes identified abnormal points, abnormal descriptions, abnormal scenarios, and abnormal results. These information together constitute the enhanced knowledge base for subsequent anomaly detection and analysis. The generation process of the enhanced knowledge base can be expressed by the formula: K = g(L original , L enhanced ) where L originalis the original log dataset, and g(·) is a function used to generate an enhanced knowledge base K containing parsed and enhanced information.

[0094] Through the above steps, the pre-trained large language model was successfully used to comprehensively parse and enhance the content of the original log sequence, generating a structured and information-rich enhanced knowledge base. This knowledge base not only contains detailed log parsing information but also supplements a large amount of abnormal log data through content enhancement, significantly improving the training effect and accuracy of subsequent anomaly detection models.

[0095] Furthermore, the following features are included in step S2: Using the pairs of normal and abnormal log sequence samples in the enhanced knowledge base, adopting the attention mechanism and encoding and decoding architecture of Transformer to deeply encode and semantically understand the log sequence sample pairs, using the cosine similarity algorithm to compare the similarity between these sample pairs, and classifying the samples into high-similarity sample pairs and low-similarity sample pairs; For the sample pairs with high similarity, further analyze the differences between the two for targeted processing in subsequent steps.

[0096] Specifically, at this stage, the pairs of normal and abnormal log sequence samples in the enhanced knowledge base will be used for preprocessing, including operations such as log cleaning, normalization, and tokenization. Then, the processed log sequences will be converted into embedding representations. A pre-trained word embedding model can be used to map each word or phrase in the log to a vector space with a fixed dimension. In addition, multi-modal embedding representations can be combined with other features of the log (such as timestamps, log levels, etc.) to more comprehensively capture the semantic information of the log. The log sequences after embedding representation will be input into the Transformer encoder. The Transformer encoder is stacked by multiple identical layers, and each layer includes a multi-head self-attention mechanism and a feed-forward neural network. The multi-head self-attention mechanism can capture the dependencies between different positions in the log sequence, thus better understanding the context information of the log. The feed-forward neural network is used to perform non-linear transformations on the features of each position to further extract the semantic features of the log. Through the Transformer encoder, a high-dimensional semantic representation of each log sequence can be obtained. For the pairs of normal and abnormal log sequence samples, their semantic representations can be compared to identify the differences and similarities between them.

[0097] These samples are classified by calculating their similarity. First, the cosine similarity is selected as the measurement criterion because it can effectively evaluate the similarity degree between texts. For any two log sequence sample pairs s i and s j , their similarity can be calculated through the cosine similarity sim(s i , s j ) as follows:

[0098]

[0099] where · represents the dot product of vectors, and ∥·∥ represents the norm of vectors. The value range of cosine similarity is from -1 to 1, and the larger the value, the more similar the two samples are.

[0100] Next, according to the set threshold θ, we divide the samples into high - similarity sample pairs S high and low - similarity sample pairs S low :

[0101] S high ={(s i , s j ) | sim(s i , s j ) > θ}

[0102] S low ={(s i , s j ) | sim(s i , s j ) ≤ θ}

[0103] This process helps us identify pairs of log sequences that appear very similar on the surface but actually have subtle yet important differences.

[0104] For high - similarity sample pairs (s i , s j ) ∈ S high , we will further analyze the differences between them. This difference analysis can help us perform more precise processing in subsequent steps. Specifically, we can define a difference function diff(·) to identify the different parts between two log sequences:

[0105]

[0106] This function is used to find the different fields or content in two log sequences, thus helping us better understand which parts are the key factors leading to anomalies.

[0107] Through the above steps, we can not only effectively distinguish high - similarity and low - similarity log sequence sample pairs, but also deeply analyze the differences between high - similarity sample pairs, providing a basis for subsequent dynamic masked language modeling.

[0108] Furthermore, the S3 includes the following features: applying a Byte-Pair Encoding (BPE) tokenizer to preprocess the samples screened in step S2 to generate tokens; differentiating normal sample tokens and abnormal sample tokens during the tokenization process so as to adopt different processing strategies for different types of samples in the subsequent steps; implementing an improved Dynamic Masked Language Modeling. For highly similar samples, identifying and masking the parts where there are differences between the abnormal log sequence and the normal log sequence; for low-similarity samples, randomly selecting logs within a set threshold range for masking replacement to increase the data diversity of model training.

[0109] Specifically, at this stage, the Byte-Pair Encoding (BPE) tokenizer will be used to preprocess the log samples screened in step S2. BPE is an efficient tokenization method that can split the text into meaningful tokens. Let BPE(·) be the Byte-Pair Encoding tokenizer, and perform tokenization on the sample s:

[0110] s′ = BPE(s)

[0111] where s′ is the token sequence after tokenization. In this way, we can convert the original log sequence into a form suitable for model processing.

[0112] During the tokenization process, normal sample tokens and abnormal sample tokens will also be differentiated. The purpose of doing this is to adopt different processing strategies for different types of samples in the subsequent steps. For example, normal sample tokens can be labeled as s normal , while abnormal sample tokens can be labeled as s anomaly . This differentiation helps the model better learn the differences between normal and abnormal logs.

[0113] Next is to implement the improved Dynamic Masked Language Modeling. For highly similar sample pairs (s i , s j ) ∈ S high , identify and mask the parts where there are differences between the abnormal log sequence and the normal log sequence. Specifically, define a masking function mask(·) to replace the different parts:

[0114]

[0115] where diff(s i , s j ) is a function used to identify the different parts between two log sequences. In this way, the BERT model can learn to identify and distinguish these key different parts during the training process, thereby improving the accuracy of anomaly detection.

[0116] For low-similarity sample pairs (s i , s j ) ∈ S low , we randomly select logs within the set threshold range for masked replacement to increase the data diversity for model training. Specifically, define a random masking function random_mask(·):

[0117]

[0118] where δ is the set threshold range. This method can help the model generalize better and avoid overfitting to specific log patterns.

[0119] In addition, to ensure that the model can fully learn the differences between normal and abnormal logs, we continuously adjust the masking strategy during training. In this way, the BERT model can not only learn how to predict the masked tokens based on the context but also improve its ability to understand the differences between normal and abnormal logs.

[0120] Furthermore, in step S4, train a BERT model dedicated to log sequence anomaly detection based on the tokens obtained from the above dynamic masking process; during training, the model learns how to predict the masked tokens (representing tokens) based on the context to improve its ability to understand the differences between normal and abnormal logs; use the trained dedicated BERT model to perform log anomaly detection and anomaly root cause tracing, not only to determine whether the log sequence is abnormal but also to locate the specific part causing the anomaly; during anomaly detection, calculate the error and probability distribution between the model prediction result and the actual log content, and perform weighted processing as the criterion for judging whether the log is abnormal.

[0121] Specifically, at this stage, train a BERT model dedicated to log sequence anomaly detection based on the tokens obtained from the dynamic masking process in step S3. Let M be the BERT model, and use the masked token sequence s masked for training:

[0122] M = train(M, s masked )

[0123] During training, the BERT model will learn how to predict the masked tokens based on the context. This self-supervised learning method helps the model better understand the subtle differences between normal and abnormal logs. For example, for a masked token t i , the model needs to predict its true value t i ′, and optimize the model parameters by minimizing the prediction error:

[0124]

[0125] where θ are the parameters of the model, and L(·,·) is the loss function, and is the token value predicted by the model.

[0126] After training is completed, the trained BERT model will be used to perform log anomaly detection and anomaly root cause tracing. Specifically, for a new log sequence s new , it can be predicted by the model:

[0127] pred = M(s new )

[0128] where pred contains the judgment on whether the log sequence is abnormal and the specific part of its abnormal root cause. To further improve the detection accuracy, during the anomaly detection process, the error and probability distribution between the model prediction result and the actual log content are calculated and weighted.

[0129] The F1 score and AUROC (Area Under the Receiver Operating Characteristic Curve) are used to evaluate the performance of the model. Specifically, for each log sequence s, the model will output an anomaly probability p(s and an anomaly loss loss(s):

[0130] abnormalprob = p(s)

[0131] abnormalloss = loss(s)

[0132] These values can be calculated by the following formula:

[0133]

[0134] where slen is the length of the log sequence, loss i and prob i are the anomaly loss and anomaly probability of the i-th token respectively.

[0135] To comprehensively evaluate the anomaly degree of the log sequence, the TOPK aggregation function is used to calculate the final anomaly score:

[0136] abnormal_loss = TOPK(scoreloss)

[0137] abnormal_prob = TOPK(scoreprob)

[0138] The TOPK function here selects the top K highest anomaly loss and anomaly probability values for weighted averaging to obtain the final anomaly score. If this score exceeds the set threshold τ, the log sequence is considered anomalous.

[0139] In addition, to more precisely locate the specific part causing the anomaly, the attention mechanism of the model can also be utilized. By analyzing the attention scores of the model at different positions, we can identify which parts contribute the most to the anomaly judgment of the model. Let A be the attention matrix of the model, and we can extract the attention scores corresponding to each token:

[0140] a i = A[i]

[0141] These scores can help us locate the specific log entry or field, thus providing more detailed information about the root cause of the anomaly.

[0142] Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed, as long as they do not depart from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the different dependent claims and the features described herein can be combined in a manner different from that described in the original claims. It should also be understood that the features described in connection with a single embodiment can be used in other described embodiments.

Claims

1. A method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling, characterized in that Including: Using a pre-trained log anomaly detection model to perform anomaly detection and anomaly root cause tracking on the log to be detected. The training process of the log anomaly detection model is as follows: Step 1: Obtain historical log samples, mark the normal and abnormal conditions of the samples, use the marked historical log samples to construct roles, tasks, and workflows with a dynamic prompt template, parse and enhance the content of each marked historical log sample using a pre-trained language model based on the roles, tasks, and workflows, and convert each sample into a positive and abnormal sample pair to generate an enhanced knowledge base; Step 2: Extract normal log sequence samples and abnormal log sequence samples from the enhanced knowledge base, and use the attention mechanism, encoding, and decoding architecture of Transformer to deeply encode and semantically understand the log sequence samples to obtain the relevance and deep semantic features between the extracted sample fields; Step 3: Use the relevance and deep semantic features between the sample fields, and adopt the cosine similarity algorithm to calculate the similarity between every two samples in the enhanced knowledge base, obtain the similarity pairs of the samples, and divide the similarity pairs into high-similarity sample pairs and low-similarity sample pairs; Step 4: Use a byte pair encoding tokenizer to preprocess the high-similarity sample pairs and low-similarity sample pairs to generate high-similarity sample pair tokens and low-similarity sample pair tokens; Step 5: Mask the high-similarity sample pair tokens with a difference-sensitive mask, and mask the low-similarity sample pair tokens with a threshold-constrained random mask; obtain the masked normal tokens and abnormal tokens; Step 6: Establish a BERT model, and use the masked normal tokens and abnormal tokens to train the BERT model to obtain a log anomaly detection model.

2. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 1, wherein In Step 1, the dynamic prompt template uses the marked historical log samples to construct roles, tasks, and workflows, inputs the roles, tasks, and workflows into the pre-trained language model to perform semantic depth parsing based on questions on the historical log samples, obtains a three-dimensional structured parsing result including an event essence layer, a system impact layer, and a structure type layer, and then uses the parsing result to generate a sample pair containing positive and abnormal samples for each sample.

3. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 2, wherein In Step 1, the enhanced knowledge base includes log samples marked with normal and abnormal conditions, parsing results, content enhancement results, unique identifiers of the logs, the essence of the events in the logs, the impact of the events on the system, the structure of the events, and the event types; The content enhancement results include identified anomaly points in the log samples, anomaly descriptions, anomaly scenarios, and anomaly results.

4. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 2, wherein The formula for the dynamic prompt template is: Prompt = Role + Task + WorkFlow + x log,label Among them, Role represents the role of the large model expert, Task represents the set of tasks that the pre-trained language model needs to execute, WorkFlow represents the issues that assist the pre-trained language model in thinking, and x log,label represents the sample log information and its positive / abnormal markings.

5. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 1, wherein In Step 3, the method for dividing the samples into high-similarity sample pairs and low-similarity sample pairs is: where · represents the dot product of vectors, and ∥·∥ represents the norm of vectors; a threshold θ is set to divide the samples into high - similarity sample pairs S high and low - similarity sample pairs S low : S high = {(s i , s j ) | sim(s i , s j ) > θ} S low = {(s i , s j ) | sim(s i , s j ) ≤ θ} where s i and s j represent the i-th sample and the j-th sample respectively.

6. The log sequence anomaly detection method based on an enhanced knowledge base and dynamic mask modeling according to claim 1, characterized in that In Step 4, the formula for the byte pair encoding tokenizer is: s′=BPE(s) where BPE(·) is the byte pair encoding tokenizer, s represents the sample, and s′ represents the token sequence after tokenization.

7. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 1, wherein In Step 5, the formula for masking the high-similarity sample tokens with a difference-sensitive mask is: Among them, diff(s i , s j ) is a function used to identify the different parts between two log sequences; represents the token after masking the highly similar samples; where x is the token in the log sequence.

8. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 4, wherein In Step 5, the formula for masking the low-similarity sample tokens with a threshold-constrained random mask is: Among them, random_mask(·) represents the random masking function, and δ is the set threshold range.

9. The method for detecting abnormal log sequences based on an enhanced knowledge base and dynamic mask modeling according to claim 5, wherein In step six, the BERT model formula is: M = train(M, s masked ) Among them, M represents the output of the BERT model, train( ) represents the model training operation, and s masked represents the masked token, including and

Images