BERT-based log anomaly monitoring method and system

Abstract

The invention provides a BERT-based log anomaly monitoring method and system, and relates to the technical field of log data monitoring, and the method comprises the steps: replacing a dynamic variable parameter through a regular expression, and preprocessing a log sequence; using BERT to extract semantic vectors, and mapping the semantic vectors to a Qwen model vector space; and training and finely tuning the Qwen model, and classifying log anomalies through the Qwen model. The method combines the semantic extraction capability of BERT and the strong representation capability of the Qwen model, and has the advantages of efficient dynamic variable processing, deep semantic extraction, accurate vector space alignment, strong anomaly detection capability and the like. Experiments show that the method is superior to the prior art in precision, recall rate and resource efficiency, and the accuracy and adaptability of log anomaly detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log data monitoring, and particularly to a log anomaly monitoring method and system based on BERT. Background Art

[0002] With the increasing complexity of software systems, log data has become an important record of the system's running state. However, log data usually contains a large amount of dynamic runtime information, making anomaly detection complex and challenging. Traditional log anomaly detection methods rely on log parsers to extract templates, but these methods perform poorly when dealing with new log messages or unstable logs and are prone to losing semantic information. In addition, existing methods often struggle to effectively process large-scale log data due to memory limitations or insufficient model performance.

[0003] In recent years, large language models (LLMs) such as BERT, GPT-4, and Qwen have made significant progress in the field of natural language processing. They have demonstrated excellent language understanding capabilities through large-scale pre-training and perform particularly well in zero-shot and few-shot tasks. However, directly applying LLMs to log anomaly detection poses challenges, such as the dynamics of log sequences, interference from variable parameters, and memory limitations of the models. Therefore, there is an urgent need for an innovative log anomaly monitoring method that can fully utilize the semantic understanding capabilities of LLMs while overcoming the limitations of traditional methods. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a log anomaly monitoring method and system based on BERT, which can significantly improve the accuracy and adaptability of log anomaly detection.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A log anomaly monitoring method based on BERT, comprising:

[0007] Identifying and replacing dynamic variable parameters in the initial log message using regular expressions to obtain a preprocessed log sequence;

[0008] Inputting the log sequence into a BERT model to encode the log message of the log sequence and obtain a semantic vector;

[0009] Mapping the semantic vector to the vector representation space of the initial Qwen model through a linear layer to obtain a projected vector representation;

[0010] Training and fine-tuning the initial Qwen model to obtain a final Qwen model;

[0011] Input the vector representation into the final Qwen large model, and classify the vector representation through the architecture based on the Transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal, and obtain the monitoring result.

[0012] Preferably, use regular expressions to identify and replace the dynamic variable parameters in the initial log message to obtain the preprocessed log sequence, including:

[0013] Use regular expressions to identify the dynamic variable parameters in the initial log message;

[0014] Replace the dynamic variable parameters with wildcards to obtain the preprocessed log sequence.

[0015] Preferably, the dynamic variable parameters include accounts, directory paths, and IP addresses.

[0016] Preferably, input the log sequence into the BERT model to encode the log message of the log sequence to obtain a semantic vector, including:

[0017] Input the preprocessed log sequence into the BERT model;

[0018] Encode the log message of the log sequence through the tokenizer of the BERT model to generate the semantic vector.

[0019] Preferably, the formula for mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer is:

[0020] V = Concat(Head1, Head2, …, Head h ) · W o + b

[0021] where, let the input semantic vector be C, with a dimension of The output projected vector representation is V, with a dimension of where, each head Head i The calculation formula is: Head i = σ(C · W i + b i ) · α i ; N is the number of log messages, d BERT is the hidden layer dimension of the BERT model, the projection weight matrix h is the number of heads of the multi-head projection, d Qwen is the hidden layer dimension of the Qwen large model, the bias term σ is the activation function, α iis the dynamic weight adjustment coefficient, and the calculation formula of the dynamic weight adjustment coefficient is: The score value score of the i-th item i The calculation formula of is: score i = mean(C·W i ); Concat(·) is the multi-head splicing function; W o is the output weight matrix, b is the output bias term,

[0022] Preferably, training and fine-tuning the initial Qwen large model to obtain the final Qwen large model includes:

[0023] Fine-tuning the initial Qwen large model to capture the answer template, and training the initial Qwen large model to respond "This sequence is abnormal / normal" to the prompt "Is this sequence normal or abnormal?";

[0024] Training the BERT model and projector to project the semantic vector of the log message into the embedding space of the fine-tuned Qwen large model;

[0025] Fine-tuning the BERT model, projector and Qwen large model to ensure that all components work together and achieve the best performance, and obtaining the final Qwen large model.

[0026] Preferably, the determination metrics for the best performance include: precision, recall, and F1 score.

[0027] A BERT-based log anomaly monitoring system includes:

[0028] A sequence preprocessing unit for identifying and replacing dynamic variable parameters in the initial log message using regular expressions to obtain a preprocessed log sequence;

[0029] A vector encoding unit for inputting the log sequence into the BERT model to encode the log message of the log sequence to obtain a semantic vector;

[0030] A vector mapping unit for mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer to obtain a projected vector representation;

[0031] A model fine-tuning unit for training and fine-tuning the initial Qwen large model to obtain the final Qwen large model;

[0032] A monitoring unit for inputting the vector representation into the final Qwen large model, classifying the vector representation through the architecture based on the Transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal, and obtaining a monitoring result.

[0033] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0034] The present invention provides a method for monitoring log anomalies based on BERT, including: identifying and replacing dynamic variable parameters in the initial log message using regular expressions to obtain a preprocessed log sequence; inputting the log sequence into the BERT model to encode the log message of the log sequence to obtain a semantic vector; mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer to obtain a projected vector representation; training and fine-tuning the initial Qwen large model to obtain a final Qwen large model; inputting the vector representation into the final Qwen large model, and classifying the vector representation through the architecture based on the Transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal and obtain a monitoring result. By combining the advantages of BERT and the Qwen large model, the present invention proposes an efficient and robust method for monitoring log anomalies. Its beneficial effects include efficient processing of dynamic variable parameters, deep extraction of semantic information, alignment of vector representation spaces, an innovative three-stage training program, efficient anomaly detection capabilities, and reduced dependence on log parsers. Experimental results show that this method is superior to the prior art in terms of accuracy, recall rate, and resource utilization efficiency, and can significantly improve the accuracy and adaptability of log anomaly detection. Description of the Drawings

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a flowchart of the method provided by the embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the technical route provided by the embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of the system structure provided by the embodiment of the present invention. Detailed Embodiments

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] The purpose of the present invention is to provide a BERT-based log anomaly monitoring method and system, which can significantly improve the accuracy and adaptability of log anomaly detection.

[0041] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0042] Figure 1 The method flow chart provided for the embodiments of the present invention is as Figure 1 shown. The present invention provides a BERT-based log anomaly monitoring method, including:

[0043] Step 100: Use regular expressions to identify and replace dynamic variable parameters in the initial log message to obtain a preprocessed log sequence;

[0044] Step 200: Input the log sequence into the BERT model to encode the log message of the log sequence to obtain a semantic vector;

[0045] Step 300: Map the semantic vector to the vector representation space of the initial Qwen large model through a linear layer to obtain a projected vector representation;

[0046] Step 400: Train and fine-tune the initial Qwen large model to obtain a final Qwen large model;

[0047] Step 500: Input the vector representation into the final Qwen large model, and classify the vector representation through the architecture based on the transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal and obtain a monitoring result.

[0048] Preferably, using regular expressions to identify and replace dynamic variable parameters in the initial log message to obtain a preprocessed log sequence includes:

[0049] Use regular expressions to identify dynamic variable parameters in the initial log message;

[0050] Replace the dynamic variable parameters with wildcards to obtain the preprocessed log sequence.

[0051] Preferably, the dynamic variable parameters include accounts, directory paths, and IP addresses.

[0052] Specifically, step 100 of this embodiment includes:

[0053] Step 101: Identify dynamic variable parameters in the initial log message using regular expressions

[0054] Dynamic variable parameters (such as accounts, directory paths, and IP addresses) are usually parts of log messages that are irrelevant to anomaly detection, but they increase the complexity of model training. Through regular expressions, these dynamic variable parameters can be efficiently identified.

[0055] Account identification: Use regular expressions to match email addresses or usernames, for example:

[0056] :regex

[0057] [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

[0058] This expression can identify standard email formats, such as user@example.com.

[0059] Directory path identification: Match file paths or URL paths, for example:

[0060] regex

[0061] (\ / [^\s\ / ]+)+

[0062] This expression can identify Linux or Windows paths, such as / home / user / documents or C:\Users\Documents.

[0063] IP address identification: Match IPv4 addresses, for example:

[0064] regex

[0065] :\d{1,3}\.){3}\d{1,3}\b

[0066] This expression can identify standard IPv4 addresses, such as 192.168.0.1.

[0067] Step 102: Replace the dynamic variable parameters with wildcards

[0068] After identifying the dynamic variable parameters, replace them with the wildcard <*>, in order to simplify the log structure and retain semantic information.

[0069] Replacement method: Through the replacement function of regular expressions (such as the re.sub method in Python), replace the matched dynamic variable parameters with <*>.

[0070] Example:

[0071] Original log message:

[0072] sql

[0073] User john.doe@example.com accessed / home / user / documents from IP 192.168.0.1.

[0074] Log message after replacement:

[0075] sql

[0076] User <*> accessed <*> from IP <*>.

[0077] This replacement method can effectively remove the interference of dynamic variables while retaining the core semantic structure of the log.

[0078] Step 103: Optimize the adaptability and efficiency of regular expressions

[0079] To ensure that regular expressions can adapt to log messages in different formats and improve processing efficiency, the regular expressions can be optimized as follows:

[0080] Unified processing of multiple types of parameters: Combine multiple regular expressions into one expression to uniformly identify dynamic variable parameters:

[0081] :regex

[0082] (? <ip>\b(?:\d{1,3}\.){3}\d{1,3}\b)|(?: <path> (\ / [^\s\ / ]+)+)|(？ <account>[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})

[0083] This expression can identify IP addresses, directory paths, and account information simultaneously.

[0084] In this embodiment, by batch-processing log messages, the number of repeated calls to the regular expression is reduced, thereby improving the processing efficiency; when a new log format appears, it can be adapted by extending the rules of the regular expression without retraining the model. Through the above three steps, this embodiment can efficiently identify and replace the dynamic variable parameters in the log messages, obtaining a preprocessed log sequence, providing a standardized input for subsequent log anomaly detection.

[0085] Preferably, input the log sequence into the BERT model to encode the log messages in the log sequence, obtaining semantic vectors, including:

[0086] Input the preprocessed log sequence into the BERT model;

[0087] Encode the log messages in the log sequence through the tokenizer of the BERT model to generate the semantic vectors.

[0088] Specifically, this embodiment takes the preprocessed log sequence after dynamic variable parameter replacement as input and passes it to the tokenizer of the BERT model. The tokenizer of the BERT model can split the text in the log sequence into subword units and generate corresponding vocabulary indices (token IDs) at the same time. During this process, the BERT tokenizer will automatically process special characters, punctuation marks, and wildcards <*>, ensuring the integrity of the log semantic structure. Then, input the vocabulary indices (token IDs) generated by the tokenizer into the BERT model, and BERT will encode the input sequence through its multi-layer Transformer architecture to generate high-dimensional semantic vectors. The encoding process of BERT will combine context information to generate a context-related representation for each token, and finally obtain the overall semantic vector of the log sequence.

[0089] Preferably, the formula for mapping the semantic vector to the vector representation space of the initial Qwen large model through a linear layer is:

[0090] V = Concat(Head1, Head2, …, Head h )·W o +b

[0091] where, let the input semantic vector be C, with a dimension of The projected vector of the output is represented as V, and its dimension is where each head i is calculated as: Head i = σ(C · W i + b i ) · α i ; N is the number of log messages, d BERT is the hidden layer dimension of the BERT model, the projection weight matrix h is the number of heads for multi-head projection, d Qwen is the hidden layer dimension of the Qwen large model, the bias term σ is the activation function, α i is the dynamic weight adjustment coefficient, and the calculation formula for the dynamic weight adjustment coefficient is: The score value score of the i-th item i is calculated as: score i = mean(C · W i ); Concat(·) is the multi-head concatenation function; W o is the output weight matrix, b is the output bias term,

[0092] Specifically, in this embodiment, through the multi-head mechanism, each head uses an independent weight matrix and bias term, which can capture different features of the log semantic vector, thereby enhancing the expression ability of the mapping. The dynamic weight adjustment coefficient α i is introduced, and the weight of each head is dynamically adjusted according to the features of the input semantic vector to ensure that the mapping result is more accurate. A non-linear transformation is introduced through the activation function σ to enhance the expression ability of the mapping. Through the multi-head mechanism and dynamic weight adjustment, the computational complexity of each head is reduced, and at the same time, the efficiency of the mapping is improved.

[0093] Preferably, training and fine-tuning the initial Qwen large model to obtain the final Qwen large model includes:

[0094] Fine-tuning the initial Qwen large model to capture the answer template, and training the initial Qwen large model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal";

[0095] Training the BERT model and the projector to project the semantic vector of the log message into the embedding space of the fine-tuned Qwen large model;

[0096] Fine-tuning the BERT model, the projector and the Qwen large model to ensure that all components work together and achieve the best performance, and obtaining the final Qwen large model.

[0097] XD-LAD is a fine-tuning-based method that utilizes BERT to extract semantic vectors from log messages and uses the Qwen large model, a model based on the transformer decoder, to classify log sequences. This method uses a projector to align the vector representation spaces of BERT and the Qwen large model. By adopting BERT, XD-LAD effectively solves the memory overflow problem caused by excessive tokens when directly using the tokenizer of the Qwen large model to tokenize the entire log sequence. Compared with other methods, XD-LAD makes full use of the capabilities of LLMs in log anomaly detection. Based on this, this embodiment adopts its fine-tuning method as the main application mode of the model.

[0098] Considering that the log message content contains variable parameters that include dynamic runtime information, which are always irrelevant to anomalies and complicate the training of deep models, a technique is needed to identify these parameters and replace them with constant tokens. Log parsers are widely adopted in log-based anomaly detection methods and seem to be a useful technique. However, existing log parsers do not always execute correctly on all log datasets and are difficult to handle out-of-vocabulary (OOV) words in new log messages, resulting in the loss of semantic information. When the logs are unstable, these parsers become increasingly ineffective over time, making it difficult to support subsequent anomaly detection.

[0099] Due to the adoption of a structured log generation process, it is easy to identify the text format of parameters representing specific objects using regular expressions. Therefore, this embodiment replaces each variable parameter, such as account, directory path, and IP address, with '<*>'. Although it is simple, this technique provides significant performance advantages. Compared with log parsers, this preprocessing technique is more effective and does not require training.

[0100] The deep model of this embodiment consists of three main components: BERT, a linear layer, and the Qwen large model. Both BERT and the Qwen large model are pre-trained LLMs. BERT is used to extract the vector representation of log messages, and the Qwen large model is used to classify log sequences. The linear layer serves as a bridge to align the vector representation spaces of BERT and the Qwen large model. It should be noted that the model of this embodiment only contains one instance of BERT and one linear layer.

[0101] BERT: BERT generates a semantic vector by processing the semantic vector of the classification token ([CLS]) and passing it through a linear layer and a tanh activation function. Each preprocessed log message is encoded into a semantic vector using the BERT tokenizer and the BERT model. For the preprocessed log sequence, the output of BERT is a sequence of semantic vectors:

[0102]

[0103] where \(N\) represents the length of the log sequence (i.e., the number of log messages), and \(d\) BERT represents the dimension of each semantic vector (i.e., the hidden size). XD-LAD uses the BERTbase model, which consists of 12 layers of Transformer encoders, and each Transformer encoder contains 768 hidden units. Therefore, \(d\) BERT is 768.

[0104] 2) Linear layer: The linear layer maps the semantic vector to the token embedding vector accepted by the Qwen large model, denoted as where \(d\) Qwen is the hidden size of Llama. The projector aims to align the vector representation spaces of BERT and Llama. To perform prompt tuning on the Qwen large model of the Transformer decoder-based LLM, this embodiment generates corresponding text queries based on the embedded log sequence. Specifically, each query consists of three components.

[0105] 3) To perform prompt fine-tuning on the large language model Llama based on the Transformer decoder, this embodiment generates corresponding text queries according to the embedded log sequence. Specifically, each query consists of three parts.

[0106] The first component introduces the log sequence. The second component includes the token embedding \(E\) output by the projector. The third component queries whether the sequence is abnormal. Where \(A\) and \(Q\) are the number of tokens generated by marking the first and third components respectively. Then, the token embeddings of the three components are concatenated. This embodiment uses Qwen2.5 - 14b to perform this process.

[0107] C. Training

[0108] Minority class oversampling: XD-LAD is a supervised anomaly detection method, which means it requires labeled normal and abnormal samples for training. However, supervised anomaly detection methods usually face the challenge of data imbalance, which can lead to bias in model training. In the anomaly detection task, there are only two classes: normal and abnormal, and the number of instances in each class is uncertain. To address data imbalance, this embodiment oversamples the class with fewer samples to ensure that the proportion of the minority class is not less than \(\beta\). Formally, assume that the proportion of the minority class is \(\alpha\), \(\alpha\lt\beta\), and the total number of samples is Sample_num. To make the proportion of the minority class reach \(\beta\), it will be oversampled to the following number:

[0109] β(1 - α)^(1 - β)×Sample_num(1)

[0110] This adjustment will make the proportion of the minority class equal to β.

[0111] Training objective: The objective of this embodiment is to train a deep model to predict whether a given log sequence is normal or abnormal. In this embodiment, the model is fine-tuned to make appropriate responses: if the sequence is abnormal, it outputs "This sequence is abnormal"; if normal, it outputs "This sequence is normal". This embodiment uses cross-entropy loss as the loss function.

[0112] Training process: To train the deep model of this embodiment, this embodiment follows three main stages.

[0113] Stage 1. Fine-tune the Qwen large model to capture the answer template:

[0114] The first stage involves fine-tuning the Qwen large model to capture the answer template. Specifically, this embodiment trains the Qwen large model to respond with "This sequence is abnormal / normal" to the prompt "Is this sequence normal or abnormal?". This stage only requires a small number of data samples.

[0115] Stage 2. Train the log message embedder:

[0116] The second stage involves training the log message embedder, specifically BERT and the projector. The goal of this stage is to project each log message into the most appropriate token embedding in the Qwen large model, enabling the Qwen large model to distinguish whether a given log sequence is normal or abnormal.

[0117] Stage 3. Fine-tune the entire model:

[0118] Finally, this embodiment fine-tunes the entire model to ensure the collaborative and accurate performance of all components.

[0119] Efficient fine-tuning of LLMs: To reduce the cost of fine-tuning LLMs (BERT and the Qwen large model) with a large number of parameters, this embodiment uses QLoRA to minimize memory usage. QLoRA achieves this by backpropagating gradients into a frozen 4-bit quantized model while maintaining the same performance level as the full 16-bit fine-tuning process.

[0120] Figure 2 Shows the architecture of the LogLLM framework, which utilizes large language models (LLMs) for log anomaly detection. It consists of three main components: BERT, the linear layer, and Qwen.

[0121] Preprocessing:

[0122] Log sequence: The input log sequence is preprocessed, and regular expressions are used to identify and replace variable parameters (such as account, directory path, and IP address) with wildcards <*>.

[0123] BERT: The preprocessed log sequence is input into the BERT model, and BERT encodes each log message into a semantic vector.

[0124] Linear layer: The projector maps the semantic vectors output by BERT to the token embedding vectors accepted by the Qwen model.

[0125] 2. Model architecture:

[0126] BERT: The BERT model is used to extract the semantic vectors of log messages, and each log message is encoded into a semantic vector.

[0127] Projector: The projector maps the semantic vectors output by BERT to the token embedding vectors accepted by the Llama model.

[0128] Llama: The Llama model is an LLM based on the transformer decoder. It receives the token embedding vectors output by the projector and outputs a judgment on whether the log sequence is abnormal.

[0129] 3. Log sequence classification:

[0130] Text query generation: The token embedding vectors output by the projector are concatenated with two text query components (introducing the log sequence and asking whether it is abnormal) to generate a text query.

[0131] Qwen: The Qwen model encodes the text query and outputs a judgment on whether the log sequence is abnormal.

[0132] To verify the superiority of this method, this embodiment compares XD-LAD with five state-of-the-art semi-supervised methods: DeepLog, Log Anomaly, PLELog, FastLogAD, and LogBERT. This embodiment also compares it with three supervised methods: LogRogobust, CNN, and NeuralLog.

[0133] In the experiments of this embodiment, the hyperparameter β is set to 30%. This embodiment uses the Adam optimizer to train the model with a mini-batch size of 32. Unless otherwise specified, the training process is configured as follows: In the first stage, only 1000 samples are involved, and the learning rate is 5e-4. The second and third stages each consist of two sub-stages, and the learning rate is 5e-5. For fair comparison, this embodiment configures the hyperparameters for all comparison methods according to the values provided in their original articles.

[0134] This embodiment uses the widely adopted precision, recall, and F1-score to evaluate the performance of these methods. The calculation methods of these metrics are as follows:

[0135]

[0136] Among them, TP, FN, and FP represent true positive, false negative, and false positive respectively

[0137] Precision refers to the percentage of anomalies correctly detected among all the anomalies identified by the model, while recall refers to the percentage of anomalies correctly identified from all the true anomalies. The F1 score combines these two metrics into a single measure, providing a balanced assessment of the model's performance in detecting anomalies.

[0138] Dataset:

[0139] To evaluate the logarithm-based anomaly detection method of this embodiment, this embodiment selects two public datasets: HDFS and Thunderbird. The detailed information of each dataset is shown below

[0140] The HDFS (Hadoop Distributed File System) dataset is generated by running Hadoop-based optimized jobs on more than 200 Amazon EC2 nodes, containing a total of 11,175,629 log messages. These log messages are grouped into different log windows according to their block_id, which reflects the program execution in HDFS. Among them, 16,838 blocks (2.93%) represent system anomalies.

[0141] Table 1 presents the experimental results of various logarithm-based anomaly detection methods conducted on the HDFS and Thunderbird datasets.

[0142] Table 1

[0143]

[0144] The proposed XD-LAD achieved the highest F1 score on all datasets. On average, the F1 score of XD-LAD was 1.7% higher than that of the existing best method, NeuralLog, demonstrating its effectiveness in log-based anomaly detection. Although LLMs were adopted in DeepLog, RAPID, NeuralLog, and Fast Anomaly Detection, their performance was still unsatisfactory. In contrast, DeepLog and Fast utilized a Transformer encoder-based model to extract semantic vectors from log messages. However, NeuralLog adopted a smaller model, while Fast used distance-based comparison for classification. On the other hand, XD-LAD utilized BERT to extract semantic vectors and the Qwen large model, a Transformer decoder-based model, for anomaly detection. The representation spaces of BERT and the Qwen large model were aligned by a projector, fully exploiting the potential of LLMs for log-based anomaly detection.

[0145] In addition, XD-LAD achieved a balance between precision and recall, indicating that it maintained a low false positive rate and minimized the false negative rate. In contrast, methods like RAPID were overly sensitive to anomalies, often resulting in a large number of false alarms. For example, on the HDFS dataset, although the precision of RAPID was 1, its recall was only 0.859, making it impractical for real-world use. Similarly, methods such as DeepLog, Log Anomaly, and LogBERT also exhibited similar problems. On the other hand, Fast was not sensitive enough to anomalies, leading to many undetected anomalies. This embodiment proposed XD-LAD, a new log-based anomaly detection framework that utilized LLMs. XD-LAD simultaneously adopted a Transformer encoder-based and a decoder-based LLM, specifically BERT and the Qwen large model, for log-based anomaly detection. BERT was used to extract semantic vectors from log messages, while the Qwen large model was used to classify log sequences. To ensure the consistency of log semantics, this embodiment introduced a projector to align the vector representation spaces of BERT and the Qwen large model. XD-LAD adopted an innovative three-stage procedure for training, aiming to improve performance and adaptability. Extensive experiments conducted on four publicly available real-world datasets showed that XD-LAD achieved remarkable performance. Subsequently, ablation studies further confirmed the effectiveness of the three-stage training procedure of this embodiment.

[0146] As Figure 3 shown, corresponding to the above methods, this embodiment also provided a BERT-based log anomaly monitoring system, including:

[0147] A sequence preprocessing unit for identifying and replacing dynamic variable parameters in the initial log messages using regular expressions to obtain a preprocessed log sequence;

[0148] A vector encoding unit, configured to input the log sequence into a BERT model to encode the log messages in the log sequence, so as to obtain semantic vectors;

[0149] A vector mapping unit, configured to map the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer, so as to obtain the projected vector representation;

[0150] A model fine-tuning unit, configured to train and fine-tune the initial Qwen large model to obtain a final Qwen large model;

[0151] A monitoring unit, configured to input the vector representation into the final Qwen large model, and classify the vector representation through the architecture based on the transformer decoder in the final Qwen large model, so as to monitor whether the log sequence is abnormal and obtain a monitoring result.

[0152] The beneficial effects of the present invention are as follows:

[0153] (1) The present invention identifies and replaces the dynamic variable parameters (such as account, directory path, and IP address) in the log message with wildcards <*> through regular expressions, avoiding the complexity and semantic information loss problems of traditional log parsers; improving the efficiency and robustness of log preprocessing, especially performing excellently when dealing with new log messages or unstable logs; eliminating the interference of dynamic variables on model training, simplifying the log structure, and retaining the core semantic information of the log at the same time.

[0154] (2) The present invention encodes the preprocessed log sequence by using a BERT model to extract the semantic vectors of the log messages, fully capturing the semantic features of the logs; compared with traditional log anomaly detection methods based on template IDs (such as LogBERT), this method can retain the semantic information of the logs, rather than relying solely on template matching; improving the understanding ability of complex log sequences, especially performing outstandingly when dealing with logs containing rich semantic information.

[0155] (3) The present invention maps the semantic vectors generated by BERT to the vector representation space of the Qwen large model through a linear layer, ensuring the semantic understanding consistency of the two models; solving the memory overflow problem caused by too many tokens when directly using the Qwen large model tokenizer to process the log sequence; improving the collaborative working efficiency of BERT and the Qwen large model, and giving full play to the advantages of both.

[0156] (4) The present invention adopts a three-stage training program (fine-tuning the Qwen large model, training the log message embedder, fine-tuning the overall model), and combines the QLoRA technology for efficient fine-tuning; by gradually optimizing the model components, it ensures the adaptability and performance of the model on different datasets; the QLoRA technology significantly reduces the memory usage and computational cost while maintaining the same performance level as full fine-tuning; experiments on four public datasets show that this method achieves a good balance between precision and recall, significantly reducing the false positive rate and false negative rate.

[0157] (5) The present invention uses the architecture of the Qwen large model based on the transformer decoder to classify the projected vector representation and monitor whether the log sequence is abnormal; Accuracy: Experiments show that when dealing with unstable logs, this method can effectively capture the semantic meaning of log messages and accurately detect anomalies; compared with traditional methods (such as DeepLog, LogBERT, etc.), the performance of this method is more stable on different datasets and is applicable to various log scenarios; compared with existing methods (such as RAPID), XD-LAD achieves a better balance between precision and recall, avoiding excessive false alarms or undetected anomalies.

[0158] (6) The present invention replaces dynamic variable parameters through regular expressions, avoiding the limitations of traditional log parsers when dealing with new log messages or out-of-vocabulary (OOV) words; improving the generality and adaptability of the method, especially performing well when facing new log formats or unstable logs; reducing the dependence on log parsers and lowering the implementation complexity.

[0159] (7) By combining the capabilities of BERT and the Qwen large model, the XD-LAD framework can efficiently process large-scale log data; the experimental results on four public datasets show that the performance of XD-LAD is superior to existing methods, especially performing outstandingly when dealing with complex and dynamic logs; this method can adapt to log data of different scales and is applicable to log anomaly detection in large-scale distributed systems.

[0160] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0161] In this article, specific examples are used to elaborate on the principles and implementation modes of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation modes and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.< / account> < / path> < / ip>

Claims

1. A BERT-based log anomaly monitoring method, characterized in that Including: Using a regular expression to identify and replace dynamic variable parameters in the initial log message, obtaining a preprocessed log sequence; Inputting the log sequence into a BERT model to encode the log messages in the log sequence, obtaining semantic vectors; Mapping the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer, obtaining a projected vector representation; Training and fine-tuning the initial Qwen large model to obtain a final Qwen large model; Inputting the vector representation into the final Qwen large model, and classifying the vector representation through the architecture based on the transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal, obtaining a monitoring result.

2. The method for monitoring log anomalies based on BERT according to claim 1, wherein, Using a regular expression to identify and replace dynamic variable parameters in the initial log message, obtaining a preprocessed log sequence, including: Using a regular expression to identify dynamic variable parameters in the initial log message; Replacing the dynamic variable parameters with wildcards to obtain the preprocessed log sequence.

3. The BERT-based log anomaly monitoring method according to claim 1, wherein The dynamic variable parameters include accounts, directory paths, and IP addresses.

4. The method for monitoring log anomalies based on BERT according to claim 1, wherein Inputting the log sequence into a BERT model to encode the log messages in the log sequence, obtaining semantic vectors, including: Inputting the preprocessed log sequence into the BERT model; Encoding the log messages in the log sequence through the tokenizer of the BERT model to generate the semantic vectors.

5. The method for monitoring log anomalies based on BERT according to claim 1, wherein The formula for mapping the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer is: V = Concat(Head1, Head2, …, Head h )·W o +b Among them, let the input semantic vector be C, with a dimension of The output projected vector representation is V, whose dimension is Among them, each head i The calculation formula is: Head i = σ(C·W i + b i )·α i ; N is the number of log messages, d BERT is the hidden layer dimension of the BERT model, the projection weight matrix h is the number of heads for multi-head projection, d Qwen is the hidden layer dimension of the Qwen large model, the bias term σ is the activation function, α i is the dynamic weight adjustment coefficient, and the calculation formula for the dynamic weight adjustment coefficient is: The score value score of the i-th item i The calculation formula is: score i = mean(C·W i ); Concat(·) is the multi-head concatenation function; W o is the output weight matrix, b is the output bias term, 6. The method for monitoring log anomalies based on BERT according to claim 1, characterized in that, Training and fine-tuning the initial Qwen large model to obtain a final Qwen large model, including: Fine-tuning the initial Qwen large model, capturing the answer template, and training the initial Qwen large model to respond to the prompt "Is this sequence normal or abnormal?" with "This sequence is abnormal / normal"; Training the BERT model and the projector to project the semantic vectors of the log messages into the embedding space of the fine-tuned Qwen large model; Fine-tuning the BERT model, the projector, and the Qwen large model to ensure that all components work together and achieve the best performance, obtaining the final Qwen large model.

7. The method for monitoring log anomalies based on BERT according to claim 6, wherein, The determination metrics for the best performance include: precision, recall, and F1 score.

8. A BERT-based log anomaly monitoring system, characterized in that, Including: A sequence preprocessing unit for using a regular expression to identify and replace dynamic variable parameters in the initial log message, obtaining a preprocessed log sequence; A vector encoding unit for inputting the log sequence into a BERT model to encode the log messages in the log sequence, obtaining semantic vectors; A vector mapping unit for mapping the semantic vectors to the vector representation space of the initial Qwen large model through a linear layer, obtaining a projected vector representation; A model fine-tuning unit for training and fine-tuning the initial Qwen large model to obtain a final Qwen large model; A monitoring unit is configured to input the vector representation into the final Qwen large model, classify the vector representation through the architecture based on the transformer decoder in the final Qwen large model to monitor whether the log sequence is abnormal, and obtain a monitoring result.

Images