A log anomaly monitoring method and system based on BERT

Abstract

The application provides a BERT-based log anomaly monitoring method and system, and relates to the technical field of log data monitoring, comprising: preprocessing log sequences by replacing dynamic variable parameters with regular expressions; extracting semantic vectors using BERT and mapping them to Qwen model vector space; training and fine-tuning the Qwen model to classify log anomalies. This method combines the semantic extraction capability of BERT and the powerful representation capability of the Qwen model, and has the advantages of efficient dynamic variable processing, deep semantic extraction, precise vector space alignment, strong anomaly detection capability, etc. Experiments show that the application is superior to the prior art in terms of precision, recall rate and resource efficiency, significantly improving the accuracy and adaptability of log anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of log data monitoring technology, and in particular to a log anomaly monitoring method and system based on BERT. Background Technology

[0002] As software systems become increasingly complex, log data has become a crucial record of system operational status. However, log data typically contains a large amount of dynamic runtime information, making anomaly detection complex and challenging. Traditional log anomaly detection methods rely on log parsers to extract templates, but these methods perform poorly when processing new or unstable log messages and are prone to losing semantic information. Furthermore, existing methods often struggle to effectively handle large-scale log data due to memory limitations or insufficient model performance.

[0003] In recent years, large-scale language models (LLMs) such as BERT, GPT-4, and Qwen have made significant progress in the field of natural language processing. They have demonstrated superior language understanding capabilities through large-scale pre-training, especially in zero-shot and few-shot tasks. However, directly applying LLMs to log anomaly detection presents challenges, such as the dynamic nature of log sequences, interference from variable parameters, and model memory limitations. Therefore, there is an urgent need for an innovative log anomaly detection method that can fully utilize the semantic understanding capabilities of LLMs while overcoming the limitations of traditional methods. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a BERT-based log anomaly monitoring method and system, which can significantly improve the accuracy and adaptability of log anomaly detection.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A BERT-based log anomaly detection method includes:

[0007] Regular expressions are used to identify and replace dynamic variable parameters in the initial log messages to obtain a preprocessed log sequence.

[0008] The log sequence is input into the BERT model to encode the log messages in the log sequence and obtain semantic vectors;

[0009] The semantic vectors are mapped to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation;

[0010] The initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model;

[0011] The vector representation is input into the final Qwen large model, and the vector representation is classified through the transformer decoder-based architecture of the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results.

[0012] Preferably, regular expressions are used to identify and replace dynamic variable parameters in the initial log message to obtain a preprocessed log sequence, including:

[0013] Use regular expressions to identify dynamic variable parameters in the initial log message;

[0014] The dynamic variable parameters are replaced with wildcards to obtain the preprocessed log sequence.

[0015] Preferably, the dynamic variable parameters include account, directory path, and IP address.

[0016] Preferably, the log sequence is input into the BERT model to encode the log messages in the log sequence to obtain a semantic vector, including:

[0017] The preprocessed log sequence is input into the BERT model;

[0018] The log messages in the log sequence are encoded using the tokenizer of the BERT model to generate the semantic vector.

[0019] Preferably, the formula for mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer is as follows:

[0020] V=Concat(Head1,Head2,…,Head h )·W o +b

[0021] Let the input semantic vector be C, with dimension ? The projected vector of the output is represented as V, with dimension 1. Among them, each head i The calculation formula for Head is: i =σ(C·W i +b i )·α i N is the number of log messages, d BERT The hidden layer dimension of the BERT model is represented by the projected weight matrix. h is the number of heads in the multi-head projection, d Qwen For the hidden layer dimension of the Qwen large model, the bias term is... σ is the activation function, α iThis refers to the dynamic weight adjustment factor, which is calculated using the following formula: The score of the i-th item i The calculation formula is: score i =mean(C·W) i Concat(·) is a multi-head concatenation function; W o To output the weight matrix, b is the output bias term.

[0022] Preferably, the initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model, including:

[0023] Fine-tune the initial Qwen model, capture the answer template, and train the initial Qwen model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal";

[0024] Train the BERT model and projector to project the semantic vectors of log messages into the embedding space of the fine-tuned Qwen large model;

[0025] Fine-tune the BERT model, projector, and Qwen large model to ensure that all components work together and achieve optimal performance, resulting in the final Qwen large model.

[0026] Preferably, the criteria for determining the best performance include: precision, recall, and F1 score.

[0027] A BERT-based log anomaly monitoring system includes:

[0028] The sequence preprocessing unit is used to identify and replace dynamic variable parameters in the initial log message using regular expressions to obtain the preprocessed log sequence;

[0029] A vector encoding unit is used to input the log sequence into the BERT model to encode the log messages in the log sequence and obtain semantic vectors;

[0030] The vector mapping unit is used to map the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation;

[0031] The model fine-tuning unit is used to train and fine-tune the initial Qwen large model to obtain the final Qwen large model;

[0032] The monitoring unit is used to input the vector representation into the final Qwen large model, and classify the vector representation through the transformer decoder-based architecture of the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results.

[0033] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] This invention provides a BERT-based log anomaly detection method, comprising: using regular expressions to identify and replace dynamic variable parameters in initial log messages to obtain a preprocessed log sequence; inputting the log sequence into a BERT model to encode the log messages in the log sequence to obtain semantic vectors; mapping the semantic vectors to the vector representation space of an initial Qwen model through a linear layer to obtain a projected vector representation; training and fine-tuning the initial Qwen model to obtain a final Qwen model; inputting the vector representation into the final Qwen model, and classifying the vector representation through a transformer decoder-based architecture in the final Qwen model to detect whether the log sequence is abnormal, thereby obtaining a monitoring result. This invention combines the advantages of BERT and the Qwen model to propose an efficient and robust log anomaly detection method. Its beneficial effects include efficient processing of dynamic variable parameters, deep extraction of semantic information, alignment of vector representation spaces, an innovative three-stage training procedure, efficient anomaly detection capabilities, and reduced dependence on log parsers. Experimental results show that this method outperforms existing technologies in terms of accuracy, recall, and resource utilization efficiency, and can significantly improve the accuracy and adaptability of log anomaly detection. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the technical route provided in the embodiments of the present invention;

[0038] Figure 3 This is a schematic diagram of the system structure provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The purpose of this invention is to provide a BERT-based log anomaly monitoring method and system, which can significantly improve the accuracy and adaptability of log anomaly detection.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a BERT-based log anomaly detection method, including:

[0043] Step 100: Use regular expressions to identify and replace dynamic variable parameters in the initial log message to obtain the preprocessed log sequence;

[0044] Step 200: Input the log sequence into the BERT model to encode the log messages in the log sequence and obtain semantic vectors;

[0045] Step 300: Map the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation;

[0046] Step 400: Train and fine-tune the initial Qwen large model to obtain the final Qwen large model;

[0047] Step 500: Input the vector representation into the final Qwen large model, and classify the vector representation through the transformer decoder-based architecture in the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results.

[0048] Preferably, regular expressions are used to identify and replace dynamic variable parameters in the initial log message to obtain a preprocessed log sequence, including:

[0049] Use regular expressions to identify dynamic variable parameters in the initial log message;

[0050] The dynamic variable parameters are replaced with wildcards to obtain the preprocessed log sequence.

[0051] Preferably, the dynamic variable parameters include account, directory path, and IP address.

[0052] Specifically, step 100 in this embodiment includes:

[0053] Step 101: Use regular expressions to identify dynamic variable parameters in the initial log message.

[0054] Dynamic variable parameters (such as account, directory path, and IP address) are often parts of log messages that are irrelevant to anomaly detection, but they increase the complexity of model training. These dynamic variable parameters can be efficiently identified using regular expressions.

[0055] Account identification: Use regular expressions to match email addresses or usernames, for example:

[0056] :regex

[0057] [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

[0058] This expression can recognize standard email formats, such as user@example.com.

[0059] Directory path recognition: Matches file paths or URL paths, for example:

[0060] regex

[0061] (\ / [^\s\ / ]+)+

[0062] This expression can recognize Linux or Windows paths, such as / home / user / documents or C:\Users\Documents.

[0063] IP address identification: Matching IPv4 addresses, for example:

[0064] regex

[0065] :\d{1,3}\.){3}\d{1,3}\b

[0066] This expression can recognize standard IPv4 addresses, such as 192.168.0.1.

[0067] Step 102: Replace dynamic variable parameters with wildcards

[0068] After identifying dynamic variable parameters, they are replaced with wildcards <*> to simplify the log structure and preserve semantic information.

[0069] Replacement method: Use regular expression replacement functionality (such as the re.sub method in Python) to replace the matched dynamic variable parameters with <*>.

[0070] Example:

[0071] Original log message:

[0072] SQL

[0073] User john.doe@example.com accessed / home / user / documents from IP192.168.0.1.

[0074] Replaced log message:

[0075] SQL

[0076] User<*>accessed<*>from IP<*>.

[0077] This replacement method can effectively remove the interference of dynamic variables while preserving the core semantic structure of the log.

[0078] Step 103: Optimize the adaptability and efficiency of regular expressions

[0079] To ensure that regular expressions can adapt to different log message formats and improve processing efficiency, regular expressions can be optimized:

[0080] Unified handling of multiple parameter types: Combining multiple regular expressions into a single expression to uniformly identify dynamic variable parameters.

[0081] :regex

[0082] (? <ip>\b(?:\d{1,3}\.){3}\d{1,3}\b)|(? <path> (\ / [^\s\ / ]+)+)|(？ <account>[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})

[0083] This expression can simultaneously identify IP address, directory path, and account information.

[0084] This embodiment improves processing efficiency by batch processing log messages, reducing the number of repeated calls to regular expressions. When new log formats appear, they can be adapted by extending the rules of the regular expressions without retraining the model. Through the above three steps, this embodiment can efficiently identify and replace dynamic variable parameters in log messages, obtaining a preprocessed log sequence, providing standardized input for subsequent log anomaly detection.

[0085] Preferably, the log sequence is input into the BERT model to encode the log messages in the log sequence to obtain a semantic vector, including:

[0086] The preprocessed log sequence is input into the BERT model;

[0087] The log messages in the log sequence are encoded using the tokenizer of the BERT model to generate the semantic vector.

[0088] Specifically, in this embodiment, the preprocessed log sequence after dynamic variable parameter replacement is used as input and passed to the BERT model's tokenizer. The BERT model's tokenizer can segment the text in the log sequence into subword units and generate corresponding token IDs. During this process, the BERT tokenizer automatically handles special characters, punctuation marks, and wildcards <*> to ensure the integrity of the log's semantic structure. Then, the token IDs generated by the tokenizer are input into the BERT model. BERT encodes the input sequence through its multi-layer Transformer architecture to generate a high-dimensional semantic vector. The BERT encoding process incorporates contextual information to generate a context-dependent representation for each token, ultimately obtaining the overall semantic vector of the log sequence.

[0089] Preferably, the formula for mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer is as follows:

[0090] V=Concat(Head1,Head2,…,Head h )·W o +b

[0091] Let the input semantic vector be C, with dimension ? The projected vector of the output is represented as V, with dimension 1. Among them, each head i The calculation formula for Head is: i =σ(C·W i +b i )·α i N is the number of log messages, d BERT The hidden layer dimension of the BERT model is represented by the projected weight matrix. h is the number of heads in the multi-head projection, d Qwen For the hidden layer dimension of the Qwen large model, the bias term is... σ is the activation function, α i This refers to the dynamic weight adjustment factor, which is calculated using the following formula: The score of the i-th item i The calculation formula is: score i =mean(C·W) i Concat(·) is a multi-head concatenation function; W o To output the weight matrix, b is the output bias term.

[0092] Specifically, this embodiment employs a multi-head mechanism, where each head uses an independent weight matrix and bias term, enabling it to capture different features of the log semantic vector, thereby enhancing the expressive power of the mapping. A dynamic weight adjustment coefficient α is introduced. i The weights of each head are dynamically adjusted based on the features of the input semantic vector to ensure more accurate mapping results. A nonlinear transformation is introduced through the activation function σ to enhance the expressive power of the mapping. Through a multi-head mechanism and dynamic weight adjustment, the computational complexity of each head is reduced, while simultaneously improving the efficiency of the mapping.

[0093] Preferably, the initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model, including:

[0094] Fine-tune the initial Qwen model, capture the answer template, and train the initial Qwen model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal";

[0095] Train the BERT model and projector to project the semantic vectors of log messages into the embedding space of the fine-tuned Qwen large model;

[0096] Fine-tune the BERT model, projector, and Qwen large model to ensure that all components work together and achieve optimal performance, resulting in the final Qwen large model.

[0097] XD-LAD is a fine-tuning-based approach that utilizes BERT to extract semantic vectors from log messages and employs the Qwen large model, a transformer-based decoder model, to classify log sequences. This method uses a projector to align the vector representation spaces of BERT and the Qwen large model. By adopting BERT, XD-LAD effectively solves the memory overflow problem caused by excessive tokens when directly using the Qwen large model's tokenizer to segment the entire log sequence. Compared to other methods, XD-LAD fully leverages the capabilities of LLMs in log anomaly detection. Therefore, this embodiment uses its fine-tuning method as the primary application of the model.

[0098] Given that log message content contains variable parameters with dynamic runtime information, which are always irrelevant to anomalies, complicating deep model training, a technique is needed to identify these parameters and replace them with constant labels. Log parsers, widely adopted in log-based anomaly detection methods, seem like a useful technique. However, existing log parsers do not always perform correctly on all log datasets and struggle to handle out-of-vocabulary (OOV) words in new log messages, leading to a loss of semantic information. As logs become unstable, these parsers become increasingly ineffective over time, making it difficult to support subsequent anomaly detection.

[0099] Because of the structured log generation process, regular expressions can easily identify the text format of parameters representing specific objects. Therefore, this embodiment replaces each variable parameter, such as account, directory path, and IP address, with '<*>'. Despite its simplicity, this technique offers significant performance advantages. This preprocessing technique is more efficient than log parsers and requires no training.

[0100] The deep model in this embodiment consists of three main components: BERT, a linear layer, and the Qwen big model. Both BERT and the Qwen big model are pre-trained LLMs. BERT is used to extract vector representations of log messages, and the Qwen big model is used to classify log sequences. The linear layer acts as a bridge, aligning the vector representation spaces of BERT and the Qwen big model. It is worth noting that the model in this embodiment contains only one BERT instance and one linear layer.

[0101] BERT: BERT generates a semantic vector by processing the semantic vector of the classification token ([CLS]) and applying it through a linear layer and the tanh activation function. Each preprocessed log message is encoded into a semantic vector using the BERT tokenizer and the BERT model. For a preprocessed log sequence, the output of BERT is a sequence of semantic vectors:

[0102]

[0103] Where N represents the length of the log sequence (i.e., the number of log messages), d BERT This represents the dimension (i.e., the hidden size) of each semantic vector. XD-LAD uses the BERTbase model, which contains 12 Transformer encoder layers, each with 768 hidden units. Therefore, d BERT It is 768.

[0104] 2) Linear layer: The linear layer will convert semantic vectors Mapped to the token embedding vector accepted by the Qwen large model, it is represented as: Where d Qwen This is the hidden size of Llama. The projector aims to align the vector representation spaces of BERT and Llama. To provide hints for tuning the Qwen large model based on the transformer decoder-based LLM, this embodiment generates corresponding text queries based on embedded log sequences. Specifically, each query consists of three components.

[0105] 3) To enable cue fine-tuning on the large language model Llama based on the Transformer decoder, this embodiment generates corresponding text queries based on the embedded log sequences. Specifically, each query consists of three parts.

[0106] The first component introduces the log sequence. The second component includes a token embedding E output by the projector. The third component queries whether the sequence is abnormal. Here, A and Q represent the number of tokens generated by the first and third components, respectively. Then, the token embeddings from the three components are concatenated. This embodiment utilizes Qwen2.5-14b to perform this process.

[0107] C. Training

[0108] Minority Class Oversampling: XD-LAD is a supervised anomaly detection method, meaning it requires labeled normal and anomalous samples for training. However, supervised anomaly detection methods often face the challenge of data imbalance, which can lead to biased model training. In anomaly detection tasks, there are only two classes: normal and anomalous, and the number of instances in each class is uncertain. To address data imbalance, this embodiment oversamples the class with fewer samples to ensure that the proportion of the minority class is not less than β. Formally, assume the proportion of the minority class is α, where α < β, and the total number of samples is Sample_num. To make the proportion of the minority class reach β, it will be oversampled to the following number:

[0109] β(1-α)1-β×Sample_num(1)

[0110] This adjustment will make the proportion of the minority class equal to β.

[0111] Training Objective: The objective of this embodiment is to train a deep model to predict whether a given log sequence is normal or abnormal. This embodiment fine-tunes the model to make an appropriate response: if the sequence is abnormal, it outputs "This sequence is abnormal"; if it is normal, it outputs "This sequence is normal". This embodiment uses cross-entropy loss as the loss function.

[0112] Training process: In order to train the deep model of this embodiment, this embodiment follows three main stages.

[0113] Phase 1. Fine-tune the Qwen model to capture the answer template:

[0114] The first stage involves fine-tuning the Qwen large model to capture answer templates. Specifically, this embodiment trains the Qwen large model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal". This stage requires only a small number of data samples.

[0115] Phase 2. Training the log message embedder:

[0116] The second phase involves training the log message embedder, specifically BERT and the projector. The goal of this phase is to project each log message onto the most suitable token embedding in the Qwen large model, enabling the Qwen large model to distinguish whether a given log sequence is normal or abnormal.

[0117] Phase 3. Fine-tuning the individual model:

[0118] Finally, this embodiment makes minor adjustments to the entire model to ensure the synergy and accurate performance of all components.

[0119] Efficient Fine-Tuning of LLMs: To reduce the cost of fine-tuning LLMs with a large number of parameters (BERT and Qwen large models), this embodiment uses QLoRA to minimize memory usage. QLoRA achieves this by backpropagating gradients to a frozen 4-bit quantized model, while maintaining the same performance level as the full 16-bit fine-tuning process.

[0120] Figure 2 This demonstrates the architecture of the LogLLM framework, which utilizes a large language model (LLM) for log anomaly detection. It consists of three main components: BERT, a linear layer, and Qwen.

[0121] Preprocessing:

[0122] Log sequence: The input log sequence is preprocessed and regular expressions are used to identify and replace variable parameters (such as account, directory path and IP address) with wildcards <*>.

[0123] BERT: The preprocessed log sequence is input into the BERT model, which encodes each log message into a semantic vector.

[0124] Linear layer: The projector maps the semantic vectors output by BERT to the token embedding vectors accepted by the Qwen model.

[0125] 2. Model Architecture:

[0126] BERT: The BERT model is used to extract semantic vectors from log messages. Each log message is encoded into a semantic vector.

[0127] Projector: The projector maps the semantic vectors output by BERT to the token embedding vectors accepted by the Llama model.

[0128] Llama: The Llama model is an LLM based on a transformer decoder. It receives the token embedding vector from the projector output and outputs a judgment on whether the log sequence is abnormal.

[0129] 3. Log sequence classification:

[0130] Text query generation: The token embedding vector output by the projector is concatenated with two text query components (introducing the log sequence and querying whether there is an anomaly) to generate a text query.

[0131] Qwen: The Qwen model encodes text queries and outputs a judgment on whether the log sequence is abnormal.

[0132] To verify the superiority of this method, this embodiment compares XD-LAD with five state-of-the-art semi-supervised methods: DeepLog, Log Anomalies, PLELog, FastLogAD, and LogBERT. This embodiment also compares it with three supervised methods: LogRogobust, CNN, and NeuralLog.

[0133] In the experiments of this embodiment, the hyperparameter β was set to 30%. This embodiment uses the Adam optimizer to train a model with a mini-batch size of 32. Unless otherwise stated, the training process is configured as follows: In the first stage, only 1000 samples are involved, with a learning rate of 5e-4. The second and third stages each consist of two stages, with a learning rate of 5e-5. For fair comparison, this embodiment configures the hyperparameters according to the values ​​provided in its original paper for all comparison methods.

[0134] This embodiment uses widely adopted metrics such as precision, recall, and F1-score to evaluate the performance of these methods. These metrics are calculated as follows:

[0135]

[0136] TP, FN, and FP represent true positive, false negative, and false positive, respectively.

[0137] Precision refers to the percentage of anomalies correctly detected out of all anomalies identified by the model, while recall refers to the percentage of anomalies correctly identified out of all true anomalies. The F1 score combines these two metrics into a single measure, providing a balanced evaluation of the model's anomaly detection performance.

[0138] Dataset:

[0139] To evaluate the logarithm-based anomaly detection method in this embodiment, two public datasets were selected: HDFS and Thunderbird. Details of each dataset are shown below.

[0140] The HDFS (Hadoop Distributed File System) dataset was generated by running optimized Hadoop-based jobs on over 200 Amazon EC2 nodes, containing a total of 11,175,629 log messages. These log messages are grouped into different log windows based on their block_id, reflecting program execution within HDFS. Of these, 16,838 blocks (2.93%) represent system anomalies.

[0141] Table 1 presents the experimental results of various logarithmic-based anomaly detection methods on the HDFS and Thunderbird datasets.

[0142] Table 1

[0143]

[0144] The proposed XD-LAD achieved the highest F1 score across all datasets. On average, XD-LAD's F1 score is 1.7% higher than the state-of-the-art method NeuralLog, demonstrating its effectiveness in log-based anomaly detection. Despite employing LLM in DeepLog, RAPID, NeuralLog, and Fast Anomaly Detection, their performance remains unsatisfactory. In contrast, DeepLog and Fast utilize a transformer-based encoder model to extract semantic vectors from log messages. However, NeuralLog uses a smaller model, while Fast uses distance-based comparisons for classification. On the other hand, XD-LAD leverages BERT to extract semantic vectors and the Qwen large model, a transformer-based decoder model, for anomaly detection. The representation spaces of BERT and the Qwen large model are aligned via projectors, fully utilizing the potential of LLM for log-based anomaly detection.

[0145] Furthermore, XD-LAD achieves a balance between precision and recall, demonstrating that it maintains a low false positive rate and minimizes the false negative rate. In contrast, methods like RAPID are overly sensitive to anomalies, often resulting in a large number of false alarms. For example, on the HDFS set, although RAPID has a precision of 1, its recall is only 0.859, making it impractical for real-world use. Similarly, methods such as DeepLog, Log Anomalies, and LogBERT exhibit similar problems. On the other hand, speed is not sensitive enough to anomalies, leading to many undetected anomalies. This embodiment proposes XD-LAD, a novel logarithmic anomaly detection framework that leverages LLMs. XD-LAD employs both transformer encoder-based and decoder-based LLMs, specifically BERT and Qwen large models, for logarithmic anomaly detection. BERT is used to extract semantic vectors from log messages, while the Qwen large model is used to classify log sequences. To ensure consistency in logarithmic semantics, this embodiment introduces a projector to align the vector representation spaces of BERT and Qwen large models. XD-LAD employs an innovative three-stage training procedure designed to improve performance and adaptability. Extensive experiments on four publicly available real-world datasets demonstrate the significant performance gains achieved by XD-LAD. Subsequent ablation studies further confirm the effectiveness of this three-stage training procedure.

[0146] like Figure 3 As shown, corresponding to the above method, this embodiment also provides a BERT-based log anomaly monitoring system, including:

[0147] The sequence preprocessing unit is used to identify and replace dynamic variable parameters in the initial log message using regular expressions to obtain the preprocessed log sequence;

[0148] A vector encoding unit is used to input the log sequence into the BERT model to encode the log messages in the log sequence and obtain semantic vectors;

[0149] The vector mapping unit is used to map the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation;

[0150] The model fine-tuning unit is used to train and fine-tune the initial Qwen large model to obtain the final Qwen large model;

[0151] The monitoring unit is used to input the vector representation into the final Qwen large model, and classify the vector representation through the transformer decoder-based architecture of the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results.

[0152] The beneficial effects of this invention are as follows:

[0153] (1) This invention identifies and replaces dynamic variable parameters (such as account, directory path and IP address) in log messages with wildcards <*> by using regular expressions, avoiding the complexity and semantic information loss problems of traditional log parsers; it improves the efficiency and robustness of log preprocessing, especially when processing new log messages or unstable logs; it eliminates the interference of dynamic variables on model training, simplifies the log structure, and retains the core semantic information of the log.

[0154] (2) This invention uses the BERT model to encode the preprocessed log sequence and extract the semantic vector of the log message to fully capture the semantic features of the log. Compared with traditional log anomaly detection methods based on template ID (such as LogBERT), this method can retain the semantic information of the log, rather than relying solely on template matching. It improves the ability to understand complex log sequences, especially when processing logs containing rich semantic information.

[0155] (3) This invention maps the semantic vectors generated by BERT to the vector representation space of the Qwen big model through a linear layer, ensuring the consistency of semantic understanding between the two models; it solves the memory overflow problem caused by too many tokens when directly using the Qwen big model tokenizer to process log sequences; it improves the collaborative working efficiency of BERT and Qwen big model and gives full play to the advantages of both.

[0156] (4) This invention employs a three-stage training procedure (fine-tuning the Qwen large model, training the log message embedder, and fine-tuning the full model) and combines it with QLoRA technology for efficient fine-tuning. By progressively optimizing model components, the adaptability and performance of the model on different datasets are ensured. QLoRA technology significantly reduces memory usage and computational costs while maintaining the same performance level as full fine-tuning. Experiments on four public datasets show that this method achieves a good balance between accuracy and recall, and significantly reduces false positives and false negatives.

[0157] (5) This invention utilizes the Qwen large model based on the transformer decoder architecture to classify the projected vector representation and monitor whether the log sequence is abnormal; Accuracy: Experiments show that this method can effectively capture the semantic meaning of log messages and accurately detect anomalies when processing unstable logs; Compared with traditional methods (such as DeepLog, LogBERT, etc.), this method has more stable performance on different datasets and is suitable for various log scenarios; Compared with existing methods (such as RAPID), XD-LAD achieves a better balance between precision and recall, avoiding too many false alarms or failure to detect anomalies.

[0158] (6) This invention replaces dynamic variable parameters with regular expressions, avoiding the limitations of traditional log parsers when processing new log messages or out-of-vocabulary (OOV) words; it improves the versatility and adaptability of the method, especially when facing new log formats or unstable logs; it reduces the dependence on log parsers and lowers the implementation complexity.

[0159] (7) By combining the capabilities of BERT and Qwen large models, the XD-LAD framework can efficiently process large-scale log data. Experimental results on four public datasets show that XD-LAD outperforms existing methods, especially in processing complex and dynamic logs. This method can adapt to log data of different scales and is suitable for log anomaly detection in large-scale distributed systems.

[0160] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0161] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / account> < / path> < / ip>

Claims

1. A log anomaly detection method based on BERT, characterized in that, include: Regular expressions are used to identify and replace dynamic variable parameters in the initial log messages to obtain a preprocessed log sequence. The log sequence is input into the BERT model to encode the log messages in the log sequence and obtain semantic vectors; The semantic vectors are mapped to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation; The initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model; The vector representation is input into the final Qwen large model, and the vector representation is classified through the transformer decoder-based architecture of the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results. The formula for mapping the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer is as follows: V=Concat(Head1,Head2,…,Head h )·W o +b Let the input semantic vector be C, with dimension ? The projected vector output is represented as V, with dimension 1. Among them, each head i The calculation formula for Head is: i =σ(C·W i +b i )·α i N is the number of log messages, d BERT The hidden layer dimension of the BERT model is represented by the projected weight matrix. h is the number of heads in the multi-head projection, d Qwen For the hidden layer dimensions of the Qwen large model, the bias term is... σ is the activation function, α i This refers to the dynamic weight adjustment factor, which is calculated using the following formula: The score of the i-th item i The calculation formula is: score i =mean(C·W) i Concat(·) is a multi-head concatenation function; W o To output the weight matrix, b is the output bias term. The initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model, including: Fine-tune the initial Qwen model, capture the answer template, and train the initial Qwen model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal"; Train the BERT model and projector to project the semantic vectors of log messages into the embedding space of the fine-tuned Qwen large model; Fine-tune the BERT model, projector, and Qwen large model to ensure that all components work together and achieve optimal performance, resulting in the final Qwen large model.

2. The log anomaly monitoring method based on BERT according to claim 1, characterized in that, Regular expressions are used to identify and replace dynamic variable parameters in the initial log message, resulting in a preprocessed log sequence, including: Use regular expressions to identify dynamic variable parameters in the initial log message; The dynamic variable parameters are replaced with wildcards to obtain the preprocessed log sequence.

3. The log anomaly monitoring method based on BERT according to claim 1, characterized in that, The dynamic variable parameters include account, directory path, and IP address.

4. The BERT-based log anomaly monitoring method according to claim 1, characterized in that, The log sequence is input into the BERT model to encode the log messages in the log sequence, resulting in a semantic vector, including: The preprocessed log sequence is input into the BERT model; The log messages in the log sequence are encoded using the tokenizer of the BERT model to generate the semantic vector.

5. The BERT-based log anomaly monitoring method according to claim 1, characterized in that, The metrics for determining the best performance include: precision, recall, and F1 score.

6. A log anomaly monitoring system based on BERT, characterized in that, include: The sequence preprocessing unit is used to identify and replace dynamic variable parameters in the initial log message using regular expressions to obtain the preprocessed log sequence; A vector encoding unit is used to input the log sequence into the BERT model to encode the log messages in the log sequence and obtain semantic vectors; The vector mapping unit is used to map the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer to obtain the projected vector representation; The model fine-tuning unit is used to train and fine-tune the initial Qwen large model to obtain the final Qwen large model; The monitoring unit is used to input the vector representation into the final Qwen large model, and classify the vector representation through the transformer decoder-based architecture in the final Qwen large model to monitor whether the log sequence is abnormal and obtain the monitoring results. The formula for mapping the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer is as follows: V=Concat(Head1,Head2,…,Head h )·W o +b Let the input semantic vector be C, with dimension ? The projected vector output is represented as V, with dimension 1. Among them, each head i The calculation formula for Head is: i =σ(C·W i +b i )·α i N is the number of log messages, d BERT The hidden layer dimension of the BERT model is represented by the projected weight matrix. h is the number of heads in the multi-head projection, d Qwen For the hidden layer dimension of the Qwen large model, the bias term is... σ is the activation function, α i This refers to the dynamic weight adjustment factor, which is calculated using the following formula: The score of the i-th item i The calculation formula is: score i =mean(C·W) i Concat(·) is a multi-head concatenation function; W o To output the weight matrix, b is the output bias term. The initial Qwen large model is trained and fine-tuned to obtain the final Qwen large model, including: Fine-tune the initial Qwen model, capture the answer template, and train the initial Qwen model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal"; Train the BERT model and projector to project the semantic vectors of log messages into the embedding space of the fine-tuned Qwen large model; Fine-tune the BERT model, projector, and Qwen large model to ensure that all components work together and achieve optimal performance, resulting in the final Qwen large model.

Images