Log anomaly detection method and system based on large model knowledge distillation

Abstract

The invention discloses a log anomaly detection method and system based on large model knowledge distillation. The method comprises the steps that a log template is obtained through a Drain algorithm, and continuous log messages are split into log sequences through a sliding window method; guiding DeepSeek-R1 to focus on the field of log anomaly detection through prompt words and samples; inputting a training set to the DeepSeek-R1, guiding through a cue word to obtain a structured anomaly analysis report generated by the DeepSeek-R1, taking the structured anomaly analysis report as a knowledge example, and enhancing the knowledge example; and training Qwen3-4B through knowledge distillation by utilizing an enhanced knowledge example, guiding the training process by a loss function matched with a learning target, finely adjusting the Qwen3-4B by adopting a LoRA method, and carrying out log anomaly detection through the finely-adjusted Qwen3-4B. According to the invention, the cost is greatly saved.

Description

Technical Field

[0001] This invention relates to the field of log anomaly detection technology, and in particular to a log anomaly detection method and system based on large model knowledge distillation. Background Technology

[0002] Log anomaly detection methods can be broadly categorized into three types. The first type comprises statistical and machine learning methods, such as LogCluster and PCA. These methods identify anomalies by mining the pattern distribution of logs, but they primarily suffer from semantic loss and insufficient modeling. The second type is based on deep learning. DeepLog pioneered the use of Long Short-Term Memory networks to predict subsequent log template IDs, while LogAnormaly improved upon this by integrating quantitative sequence features. LogBERT and LAnoBERT employ the BERT architecture to predict masked log template IDs. These methods typically require extensive training from scratch, making training difficult, and their fixed vocabulary space limits their flexibility in adapting to new log datasets.

[0003] Existing log anomaly detection methods based on large models inevitably incur significant costs due to excessive reliance on them. For example, the LogGPT algorithm uses specific task cues to query LLMs (Log Sequence Management Models) to identify log sequence anomalies, resulting in substantial overhead. Conversely, designing algorithms to reduce the use of large models leads to incomplete coverage; for instance, the LLMeLog algorithm struggles to capture low-frequency but critical anomaly patterns. Therefore, balancing cost and accuracy is crucial in log anomaly detection methods and systems based on large models. Summary of the Invention

[0004] This invention addresses the cost-accuracy balance problem in log anomaly detection methods and systems based on large models, proposing a log anomaly detection method and system based on knowledge distillation of large models. After preprocessing, this method guides DeepSeek-R1 to focus on log anomaly detection using designed prompts and samples, and then provides the dataset to DeepSeek-R1 to generate knowledge examples, which are then optimized and enhanced. To control the cost of using large models, this invention utilizes knowledge distillation to train a lightweight open-source model, Qwen3-4B, with enhanced knowledge examples, enabling it to outperform DeepSeek-R1 in log anomaly detection. Thus, by only using the low-cost Qwen3-4B, the effect of using DeepSeek-R1 can be achieved, significantly reducing costs.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: This invention proposes a log anomaly detection method based on large model knowledge distillation, comprising: Step 1: Use the Drain algorithm to obtain the log template and use the sliding window method to split the continuous log messages into log sequences, and use a log sequence as a sample; Step 2: Guide DeepSeek-R1 to focus on log anomaly detection using designed prompts and samples; Step 3: Input the training set into DeepSeek-R1 and guide it with the prompt words to obtain a structured anomaly analysis report generated by DeepSeek-R1. Use the anomaly analysis report as knowledge examples for subsequent knowledge distillation and enhance the knowledge examples; the training set consists of multiple labeled samples. Step 4: Use DeepSeek-R1 as the teacher model and Qwen3-4B as the student model. Train Qwen3-4B using augmented knowledge examples and knowledge distillation. The training process is guided by a loss function that matches the learning objective. At the same time, use the LoRA method to fine-tune Qwen3-4B and use the fine-tuned Qwen3-4B to perform log anomaly detection.

[0006] Furthermore, in step 2, the prompt words include a task description, a formatted statement, and human knowledge injection; the task description is used to inform DeepSeek-R1 that log anomaly detection is a binary classification problem and to explain the occurrence of the anomaly; the formatted statement is used to request the response to be output in JSON format; and the human knowledge injection is used to introduce labeled log examples.

[0007] Furthermore, the training set is obtained by continuously sampling a public log dataset.

[0008] Furthermore, the anomaly analysis report includes log sequence ID, anomaly summary, key events, potential impact, and handling recommendations.

[0009] Furthermore, in step 3, the enhancement of the knowledge examples includes: Perform language validation on log anomaly detection descriptions, retaining only English descriptions; By leveraging the intermediate states and output structure features of DeepSeek-R1 in a single inference iteration, combined with a pre-defined domain rule base, the confidence score of knowledge examples is calculated, and the knowledge examples are filtered based on the confidence score.

[0010] Furthermore, the confidence score of the knowledge example is calculated as follows: In the formula, in This represents the confidence score of knowledge example x. , and These are weighting coefficients. The score for matching the domain rules of x. This represents the DeepSeek-R1 internal confidence score for x. The structural integrity score for x is calculated, where n represents the number of key tokens. Represents the set of key tokens. Indicates that in the given preceding context Under the given conditions, the model predicts the probability of the i-th key token.

[0011] Furthermore, in step 4, the loss function is: In the formula, in, It is the total loss function. It is a loss of structural consistency. It is semantic distillation loss. It is a task-oriented loss. These are weight parameters. For indicator functions, To analyze the failure penalty item, , K is a learnable parameter, representing the total number of predefined required fields. This indicates the k-th required field. This represents the output of the student model. These are the semantic vectors obtained by mean pooling from the last hidden state of the student model and the teacher model, respectively. It is the cosine similarity function. These are the probability distributions of the event → impact → suggestion causal chain within the student model and the teacher model, respectively. Denotes KL divergence, and It is an adjustable weighting coefficient. These represent the sets of domain terms extracted from the outputs of the student model and the teacher model, respectively. Used to calculate terminology consistency scores based on domain dictionary This represents the probability distribution of anomaly categories predicted by the student model for the current input log sequence. It is a high-confidence prediction of the teacher model validated by Log-Filter. It is an adjustable weighting coefficient. This is used to ensure that the student model's scoring and ranking on abnormal and normal samples are consistent with the teacher's. These represent the anomaly scores assigned to the input sequence by the student model and the teacher model, respectively.

[0012] Furthermore, the fine-tuning of Qwen3-4B using the LoRA method includes: Load the pre-trained Qwen3-4B model weights as the base model, freeze all its original parameters, and then load the pre-trained weight matrix. The update amount of the pre-trained weight matrix is ​​constrained to a low-rank decomposition form: ,in , It is a predefined low-rank, significantly smaller than Dimensions and ; The knowledge examples enhanced in step 3 are used as training data. Each training sample consists of a log sequence and structured knowledge. The log sequence and structured knowledge are concatenated into an instruction fine-tuning format according to the prompt response template, which serves as the model input and expected output. Add a step-by-step instruction hierarchy to guide the model's analysis according to the process, requiring results to be output in JSON format. As the optimization objective, the AdamW optimizer is adopted, and gradient updates are performed in 4-bit quantization mode. During fine-tuning, only the matrix is ​​updated. and ,Keep Freeze the memory and accelerate computation using the Unsloth framework; Will Low-rank update volume with LoRA Add them together to generate an independent, lightweight log anomaly detection model.

[0013] Another aspect of this invention proposes a log anomaly detection system based on large model knowledge distillation, comprising: The preprocessing unit is used to obtain log templates using the Drain algorithm and to split continuous log messages into log sequences using the sliding window method, taking a log sequence as a sample. The domain knowledge injection unit is used to guide DeepSeek-R1 to focus on the domain of log anomaly detection through designed prompts and samples; The knowledge example generation unit is used to input the training set into DeepSeek-R1, guide it with the prompt words, and obtain a structured anomaly analysis report generated by DeepSeek-R1. The anomaly analysis report is used as a knowledge example for subsequent knowledge distillation, and the knowledge example is enhanced. The training set consists of multiple labeled samples. The model training and detection unit uses DeepSeek-R1 as the teacher model and Qwen3-4B as the student model. It trains Qwen3-4B using augmented knowledge examples and knowledge distillation. The training process is guided by a loss function that matches the learning objective. At the same time, the LoRA method is used to fine-tune Qwen3-4B, and the fine-tuned Qwen3-4B is used for log anomaly detection.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a log anomaly detection method and system based on knowledge distillation from a large model. After preprocessing, the invention guides DeepSeek-R1 to focus on log anomaly detection using designed prompts and samples, and then provides the dataset to DeepSeek-R1 to generate knowledge examples, which are then optimized and enhanced. To control the cost of using a large model, this invention utilizes knowledge distillation to train a lightweight open-source model, Qwen3-4B, with enhanced knowledge examples, enabling it to outperform DeepSeek-R1 in log anomaly detection. Thus, by only calling the low-cost Qwen3-4B, the effect of using DeepSeek-R1 can be achieved, significantly reducing costs.

[0015] This invention completes log structuring and knowledge distillation preparation through preprocessing and domain knowledge injection, and then utilizes knowledge example generation and model training to achieve detection results superior to large models with hundreds of billions of parameters using only a lightweight 4B-parameter model. This invention explores a new technical path for the field of log anomaly detection that is economical, advanced, and scientific. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a log anomaly detection method based on large model knowledge distillation, provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of the prompt word design structure provided in an embodiment of the present invention; Figure 3 Examples of prompt words provided in embodiments of the present invention; Figure 4 This is a schematic diagram of a prompt response template provided in an embodiment of the present invention; Figure 5This is a schematic diagram of the architecture of a log anomaly detection system based on large model knowledge distillation, provided for an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: This invention proposes a log anomaly detection method based on knowledge distillation from a large model. It utilizes the pre-trained large language model DeepSeek-R1 and the lightweight open-source model Qwen3-4B, aiming to leverage the knowledge of DeepSeek-R1 to improve the performance of Qwen3-4B. First, the Drain algorithm is used to obtain log templates, and a sliding window method is used to split continuous log messages into log sequences. Second, designed prompts and samples guide DeepSeek-R1 to focus on log anomaly detection. Then, a training set is input into DeepSeek-R1 to obtain knowledge examples generated by DeepSeek-R1, which are then reinforced. Finally, the enhanced knowledge examples are used to train Qwen3-4B, guided by a loss function matched to the learning objective. This results in a lightweight model, Qwen3-4B, that surpasses DeepSeek-R1's capabilities in log anomaly detection. Ultimately, Qwen3-4B can detect various types of logs.

[0018] The workflow of this invention is as follows: Figure 1 As shown, the process includes four steps: preprocessing, domain knowledge injection, knowledge example generation, model training, and detection. Through this collaborative optimization process, a high-performance, low-overhead log anomaly detection method is ultimately achieved.

[0019] (1) Preprocessing: This invention employs the classic log parsing technique, the Drain algorithm. Drain effectively extracts templates from log messages and maps each log message to a corresponding template. Then, a sliding window method is used to split consecutive log messages into log sequences, with each log sequence serving as a sample.

[0020] (2) Domain knowledge injection. Domain knowledge injection is the basis of knowledge distillation. It guides DeepSeek-R1 to focus on the domain of log anomaly detection through designed prompts and samples.

[0021] (3) Knowledge Example Generation: After passing through DeepSeek-R1, the high-level reasoning process of DeepSeek-R1 can be extracted from the log dataset. At the same time, we enhance these reasoning processes to generate high-quality knowledge examples. Specifically, we input the training set (obtained from the log dataset) into DeepSeek-R1, guide it with the prompt words, and obtain a structured anomaly analysis report generated by DeepSeek-R1. The anomaly analysis report is used as a knowledge example for subsequent knowledge distillation, and the knowledge examples are enhanced. The training set consists of multiple labeled samples.

[0022] (4) Model training and detection: DeepSeek-R1 was used as the teacher model, and Qwen3-4B was used as the student model. Qwen3-4B was trained using augmented knowledge examples and knowledge distillation. This training was guided by a loss function that matched the learning objective, and the advanced LoRA method was used to fine-tune Qwen3-4B at low cost and high efficiency. Log anomaly detection was then performed using the fine-tuned Qwen3-4B.

[0023] The method specifically includes: 1. Preprocessing Preprocessing involves converting these unstructured log messages into structured log sequences. This step is auxiliary; this invention does not propose a new method for log preprocessing, but rather borrows the classic log parsing technique, the Drain algorithm. Preprocessing mainly consists of two steps: (1) Mapping each log message to a corresponding template using the Drain algorithm. (2) Splitting consecutive log messages into log sequences using a sliding window method. Referring to the experience of numerous log anomaly detection algorithms, the sliding window is set to 10, and the step size is set to 5, meaning that every 10 consecutive log messages form a log sequence, and a log sequence is a sample.

[0024] 2. Domain Knowledge Injection Domain knowledge injection is intended to guide DeepSeek-R1's development towards log anomaly detection. To enable DeepSeek-R1 to produce more detailed and accurate output, we need to design a suitable set of prompt words and training samples. The prompt word design structure is as follows: Figure 2 As shown, it consists of three parts: task description, format statement, and human knowledge injection.

[0025] (1) The interpretability of the results is ensured by the task description. The task description tells DeepSeek-R1 log anomaly detection that it is a binary classification problem and also prompts it to explain the occurrence of anomalies.

[0026] (2) Use format statements to ensure the stability of the returned results. The response must be output in JSON format to prevent some unexpected responses, which may lead to inaccurate or unreasonable results.

[0027] (3) Injecting domain knowledge with human knowledge, which refers to labeled log examples, i.e., the training samples mentioned later, which come from publicly available log anomaly detection datasets. The ultimate goal is to guide large models to generate detection results that are highly accurate, highly interpretable, and easy to automate.

[0028] Training samples are crucial, acting as a catalyst. To this end, we manually selected thirty anomalous samples and twenty normal samples from widely used public log datasets such as BGL, Spirit, and Thunderbird to form the training samples. Taking BGL as an example, we can identify representative anomalous and normal samples with strong coverage, including classic anomalous samples such as persistent errors, single critical errors, specific error sequences, and performance metric anomalies, as well as classic normal samples such as a system successfully starting up and entering standby mode, and a complete and successful user request processing flow. A high-quality, small sample set can effectively guide DeepSeek-R1 towards the field of log anomaly detection.

[0029] 3. Knowledge Example Generation Knowledge example generation is a crucial step in this invention, as the quality of the knowledge examples largely determines the learning performance of the student model. First, a training set needs to be selected from the log dataset. Second, a cue word needs to be designed for DeepSeek-R1 to generate knowledge examples. This step primarily aims to standardize the format of the knowledge examples and obtain the reasoning process of DeepSeek-R1, i.e., the thought chain (CoT) interpretation. Finally, the knowledge examples need to be optimized and enhanced. This step mainly involves filtering out low-quality generated knowledge examples. In summary, knowledge example generation mainly includes three specific processes: training set creation, cue word design, and knowledge example enhancement.

[0030] 3.1 Training Set Creation The basic method for creating the training set in this invention is as follows: First, log events are converted into log sequences using a time window method. Then, the first 10% of the log sequences are used to form the training set, and the remaining 90% are used to form the test set. Taking the widely recognized and used public log dataset BGL as an example, this demonstrates how to create the training set. Table 1 provides detailed information about the BGL dataset. For log events, we use a sliding window with a stride of 5 and a window length of 10 to segment them into log sequences.

[0031] Table 1. Detailed information about the dataset We selected 10% of all log sequences from BGL to form the training set, resulting in a training set of approximately 27,000 sequences. Considering that randomly selecting samples from the dataset for training might introduce unfair bias due to data leakage, a continuous sampling strategy was adopted for the training set to mitigate this. Specifically, the first 10% of log sequences formed the training set, and the remaining 90% formed the test set. This method makes the invention more closely resemble a real-world environment, ensuring that the training process accurately reflects the actual production environment and reducing the risk of data leakage. Although there are many types of logs, this invention is applicable to the creation of training sets for all log types.

[0032] 3.2 Prompt Design The design of prompt words is the core of knowledge example generation and determines the quality of the knowledge examples. When applied to log anomaly detection, especially on specialized datasets like BGL, DeepSeek-R1 may generate analyses that appear reasonable on the surface but are actually incorrect, potentially producing poor-quality or even erroneous knowledge examples. This can severely impact subsequent knowledge distillation processes.

[0033] Here is an example of a flawed description generated by LLM for a BGL log sequence: "A 'Sub-clock BusInversion' (SCBI) anomaly was detected, triggered by a sequence of routine RAS KERNEL INFO messages indicating successful node card initialization. When these INFO messages are processed by the compute node's monitoring agent in a specific millisecond-aligned sequence, they bypass the ECC memory protection layer, directly causing a cascading bit-flip in the non-adjacent Torus network processor, leading to a critical data corruption failure." DeepSeek-R1 invented a seemingly technical but actually non-existent anomaly type: Sub-clock BusInversion (SCBI); the description incorrectly links a software-level event to directly circumventing a hardware-level protection mechanism; and it constructs an illogical causal chain, concluding that a series of successful, informative (INFO) log messages were the direct cause of catastrophic physical hardware failure. This violates the fundamental principle that information logs describe system state, not actively cause physical damage. Hallucinations are an inherent limitation of DeepSeek-R1, and the way DeepSeek-R1 works makes the problem of hallucinations difficult to eradicate. Therefore, this invention includes necessary prompts and limitations in the prompts, specifically as follows: Figure 3 As shown.

[0034] In the design of prompts for log anomaly detection, this invention mandates a specific format for the anomaly analysis report, requiring it to include five parts: log sequence ID, anomaly summary, key events, potential impact, and handling suggestions. This forces the model to generate a thought process that conforms to the logic of log anomaly detection. Furthermore, to obtain high-quality examples of prompts, we systematically generate and filter them through the following three steps: (1) Initial sample collection: Select several real abnormal log sequences from the BGL log dataset, such as account lockout events after multiple login failures, as the basic input.

[0035] (2) Model-guided generation: Input the above log sequence into DeepSeek-R1, generate multiple sets of original analysis reports under the initial prompt word framework, and retain the candidate outputs that are semantically reasonable and conform to common sense in the domain.

[0036] (3) Manual verification and structured reconstruction: Domain experts review the authenticity, logic and compliance of candidate outputs, remove content that is illusory or erroneous, and rewrite it in accordance with the five-part structure and JSON format specified in this invention to finally form a high-quality example.

[0037] The prompts explicitly require the model to output in a standardized JSON format, effectively constraining the scope and structure of the generated content and reducing the negative impact of DeepSeek-R1 illusions or arbitrary wording on the reliability and operability of the analysis results. This multifaceted prompting engineering design strategy aims to effectively mitigate the impact of DeepSeek-R1 illusions on the quality of generated anomaly descriptions, ensuring that the output is both fact-based and structurally consistent, laying a solid foundation for subsequent knowledge distillation.

[0038] 3.3 Enhanced Knowledge Examples Knowledge example enhancement is the process of filtering generated knowledge examples. While DeepSeek-R1 boasts excellent performance and can generate high-quality knowledge examples through prompt word design, it inevitably generates some low-quality examples due to issues such as DeepSeek-R1 illusions. To filter out these low-quality knowledge examples, we designed the Log-Filter algorithm to achieve the goal of knowledge example enhancement.

[0039] The Log-Filter algorithm consists of three steps. First, language verification is performed. Since log anomaly detection descriptions are primarily in English, but DeepSeek-R1 may generate descriptions in other languages ​​in practice, only English descriptions are retained to ensure compatibility with subsequent analysis. Next is confidence scoring. This invention proposes a Lightweight Self-Feedback Confidence Scoring (LSFC) mechanism. This mechanism does not require calling additional language models or multiple generation processes; it only utilizes the intermediate states and output structure features of DeepSeek-R1's single inference iteration, combined with a pre-defined domain rule base, to achieve efficient, low-cost, and high-precision knowledge example quality assessment. The confidence scoring function... The definition is as follows: The initial weights =0.5, =0.3, =0.2.

[0040] 1. The domain rule matching score is calculated by performing pattern matching on the generated text x (i.e., knowledge example) based on a pre-built causal rule engine for operational events. Example rules include: if "FATAL" level logs are included, a restart or rollback recommendation must exist; network anomaly summaries must be associated with at least one log event containing "timeout" or "connection refused," etc. The calculation method is as follows: 2. Representing the teacher model's internal confidence score, the average probability of key decision points (such as anomaly category, action verb, and scope of impact) is extracted using the token-level prediction probability sequence generated by DeepSeek-R1 when x is generated. This step directly reuses the teacher model's internal probability distribution during inference as a quality signal, eliminating the need for secondary calls or external evaluators, thus achieving a self-feedback mechanism with zero additional overhead. The calculation method is as follows, where KeyTokens are automatically located by the placeholder positions defined in the prompt (e.g., ...).<ANOMALY_TYPE> , <action>(), no additional annotation is required.

[0041] Where n represents the number of key tokens. Represents the set of key tokens. Indicates that in the given preceding context Under the given conditions, the model predicts the probability of the i-th key token.

[0042] 3. This involves calculating a structural integrity score, the purpose of which is to check whether the output x conforms to the completeness and type validity of the required fields in the predefined JSON Schema. This part is related to the loss function. This forms a closed loop between the front-end and back-end, strengthening format consistency constraints. The calculation method is as follows: Finally, we set a threshold for the overall confidence level. To ensure that the quality of the obtained knowledge examples is as high as possible, we set the threshold to 0.85. When it is less than this threshold, the corresponding knowledge examples are filtered out.

[0043] 4. Model Training Model training is the final step in this invention, aiming to inject the capabilities of DeepSeek-R1 in log anomaly detection into Qwen3-4B through knowledge distillation. This section details how to train the Qwen3-4B model, specifically in two parts: loss function design and fine-tuning based on LoRA technology.

[0044] 4.1 Loss Function Design To effectively guide the student model Qwen3-4B during knowledge distillation, loss functions typically employ only KL divergence or cross-entropy to align the class probability distributions of the teacher and student models. However, the log anomaly detection task addressed in this invention is highly structured and semantically dependent. Directly using traditional loss functions would lead to problems such as structure mismatch and semantic drift. Therefore, this invention designs a Structured Knowledge Distillation Loss (SKDL) function, which consists of a weighted sum of three sub-losses, specifically including: 1. Structural consistency loss This constraint is used to ensure that the output of Qwen3-4B strictly adheres to a predefined JSON format template, including required fields such as "Anomaly Summary," "Key Events," "Potential Impact," "Handling Recommendations," and "Log Sequence ID." Any missing field, incorrect format, or parsing failure will be penalized, thus ensuring the structural stability of the output and compatibility with automated processing. The calculation method is as follows: in: This is an indicator function that penalizes the function if a field is missing. To resolve the failure penalty term, the structural offset is calculated based on AST alignment; , These are learnable parameters, initially set to 0.6 and 0.4, and dynamically decay with each training round.

[0045] 2. Semantic distillation loss By calculating the similarity between the Qwen3-4B output and the reinforced knowledge examples generated by DeepSeek-R1 in the semantic embedding space, Qwen3-4B is guided to learn the reasoning logic, causal chains, and domain terminology expressions of DeepSeek-R1. This loss function ensures that Qwen3-4B not only imitates DeepSeek-R1 in format but also inherits its advanced reasoning capabilities at the semantic level. The calculation method is as follows: in The semantic vectors obtained by mean pooling the last hidden state of the student model and the teacher model; The probability distribution of the event → impact → suggested causal chain generated within the model can be extracted using attention heads; Calculate terminology usage consistency scores based on domain dictionary; , Initially set to 0.3 and 0.2, dynamically adjusted according to the F1-score of the validation set.

[0046] 3. Task-oriented losses This loss is used to align the final detection decision of Qwen3-4B with the true anomaly label (or the authoritative judgment of DeepSeek-R1), preventing deviation from the actual detection target during semantic imitation. This loss ensures that the model maintains its accurate ability to distinguish anomalies while inheriting the semantic capabilities of DeepSeek-R1. This invention designs a dual-supervision mechanism: in This represents the probability distribution of anomaly categories predicted by the student model for the current input log sequence; The teacher model's high-confidence predictions need to be validated using a Log-Filter. Ensure that the student model's scoring and ranking on abnormal and normal samples are consistent with the teacher's, but allow for small deviations to retain independent judgment ability; It starts at 0.5 and decreases linearly to 0.1 as the epoch increases.

[0047] The above three sub-losses are weighted according to the parameters The weighted summation forms the final loss function: .in, The initial values ​​are set to 0.4, 0.4, and 0.2, and can be dynamically adjusted based on the performance of the validation set. This composite loss function collaboratively optimizes Qwen3-4B from three dimensions: structure, semantics, and task, enabling it to achieve detection accuracy and interpretability exceeding DeepSeek-R1 while maintaining lightweight design.

[0048] 4.2 Fine-tuning based on LoRA technology Throughout the fine-tuning process, this invention selected Qwen3-4B as our base model, a lightweight and flexible LLM. Qwen3-4B is characterized by its efficient architecture design and training strategies, achieving comprehensive performance close to or even surpassing larger models with only 4 billion parameters. It employs Grouped Query Attention (GQA) to significantly reduce inference latency and memory usage while maintaining multi-head attention capabilities. It supports context windows of up to 32,768 tokens and achieves linearly controllable memory growth through dynamic caching and sliding window mechanisms. During the pre-training phase, the model incorporates high-quality multilingual and code corpora with over a trillion tokens, and the fine-tuning phase enhances its tool invocation, logical reasoning, and complex task decomposition capabilities. Compared to traditionally powerful models like Mistral and Llama3-8B, Qwen3-4B offers significant advantages in response speed, accuracy, and memory efficiency, making it more suitable for real-world production deployment needs. In conclusion, Qwen3-4B is more suitable for our task scenario than other popular LLMs.

[0049] The core task of fine-tuning algorithms is to adapt pre-trained large models to specific downstream tasks at a lower cost, avoiding repeated training on the entire dataset. A key reason for the hallucination problem in log anomaly detection tasks is the lack of relevant domain knowledge in LLMs. Therefore, we choose to inject structured domain expertise extracted in Part 3 into large models through fine-tuning. To efficiently inject domain expertise into the model, this invention uses a 4-bit quantization fine-tuning scheme based on LoRA (Low-Rank Adaptation) to fine-tune Qwen3-4B, and optimizes Qwen3-4B using the Unsloth framework. Unsloth, as an efficient fine-tuning framework, significantly reduces GPU memory requirements and improves training speed by 2-5 times through manually optimized GPU kernel computation and gradient checkpoint compression techniques. Regarding the parameter update strategy, LoRA injects a low-rank decomposition matrix, fine-tuning only 0.1%-3% of the parameters of the target module, rather than adjusting the entire set, as detailed below: 1. Load the pre-trained Qwen3-4B model weights as the base model and freeze all its original parameters. For the pre-trained weight matrix... LoRA constrains its update amount to a low-rank decomposition form: ,in ,here It is a predefined low-rank matrix, significantly smaller than the dimension of the original matrix. and .

[0050] 2. The knowledge examples enhanced by Log-Filter are used as training data. Each sample consists of a log sequence {log_sequence} and structured knowledge {Structured_knowledge}. According to... Figure 4 The provided prompt response template concatenates the log sequence with structured knowledge into a fine-tuned instruction format, serving as both model input and expected output. To make Qwen3-4B's responses more specific and organized, we added a step-by-step instruction hierarchy to guide the model through the analysis according to a predetermined process. To make Qwen3-4B's responses more concise and structured, we strengthened output constraints, requiring the analysis results to be output in a strict JSON format.

[0051] 3. Using the aforementioned SKDL loss function as the optimization objective, the AdamW optimizer is employed, and gradient updates are performed in 4-bit quantization mode. During fine-tuning, only the matrix is ​​updated. and Maintain original weights Freeze, the forward computation of the original model becomes: This method leverages the Unsloth framework for memory compression and computational acceleration. Compared to other efficient fine-tuning methods such as hierarchical prompt injection in P-Tuning v2 or quadratic quantization in QLoRA, this approach maintains the model's inference capabilities while better adapting to the end-to-end optimization requirements of generative tasks. As a result, the fine-tuned Qwen3-4B exhibits stronger output consistency in instruction-following tasks, and its response format is naturally normalized through Dynamic Gradient Alignment, avoiding the structural biases introduced by traditional classification heads, making it more suitable for our log anomaly detection task scenario.

[0052] 4. After training, the original model weights are compared with the LoRA adapter matrix (i.e., ... The models are merged to generate a lightweight log anomaly detection model that can be used for detection. This trained log anomaly detection system does not require additional reliance on the DeepSeek-R1 model or external knowledge bases during the inference phase, and can independently complete end-to-end log anomaly detection and structured report generation.

[0053] 5. Testing During the detection phase, a finely tuned Qwen3-4B model is used to detect anomalies in the log sequences recorded by the target system. First, preprocessing converts the logs into event sequences. Then, the structured log sequences are input into the finely tuned Qwen3-4B model. Based on the structured reasoning capabilities learned during training, the model automatically generates an anomaly detection report conforming to a preset JSON schema. The detection results are output in structured JSON format, which can be directly parsed by the operations and maintenance system, alarm platform, or automated repair system. If the detection result determines an anomaly, the system can automatically trigger alarms, generate work orders, or perform root cause analysis, achieving closed-loop operations and maintenance.

[0054] This stage eliminates the need to call the original large model DeepSeek-R1, relying only on the lightweight Qwen3-4B, reducing inference costs by more than 90%, while maintaining or even surpassing the detection accuracy and interpretability of the original large model, truly realizing a "low-cost, high-performance, industrial-grade usable" log anomaly detection system.

[0055] Based on the above embodiments, such as Figure 5 As shown, this invention also proposes a log anomaly detection system based on large model knowledge distillation, comprising: The preprocessing unit is used to obtain log templates using the Drain algorithm and to split continuous log messages into log sequences using the sliding window method, taking a log sequence as a sample. The domain knowledge injection unit is used to guide DeepSeek-R1 to focus on the domain of log anomaly detection through designed prompts and samples; The knowledge example generation unit is used to input the training set into DeepSeek-R1, guide it with the prompt words, and obtain a structured anomaly analysis report generated by DeepSeek-R1. The anomaly analysis report is used as a knowledge example for subsequent knowledge distillation, and the knowledge example is enhanced. The training set consists of multiple labeled samples. The model training and detection unit uses DeepSeek-R1 as the teacher model and Qwen3-4B as the student model. It trains Qwen3-4B using augmented knowledge examples and knowledge distillation. The training process is guided by a loss function that matches the learning objective. At the same time, the LoRA method is used to fine-tune Qwen3-4B, and the fine-tuned Qwen3-4B is used for log anomaly detection.

[0056] In summary, this invention achieves high accuracy and strong interpretability in log anomaly detection while controlling computational costs and deployment overhead. Existing solutions, while directly calling large models like GPT, offer high accuracy but incur enormous costs; traditional deep learning models are low-cost but suffer from poor generalization and weak semantic understanding; and existing knowledge distillation methods are mostly used for classification tasks and cannot transfer structured reasoning capabilities. Therefore, this invention proposes a log anomaly detection method and system based on knowledge distillation from large models. It completes log structure processing and knowledge distillation preparation through preprocessing and domain knowledge injection, and then utilizes knowledge example generation and model training. This achieves superior detection performance compared to large models with hundreds of billions of parameters using only a lightweight 4B-parameter model, exploring a new technical path for log anomaly detection that is economical, advanced, and scientifically sound.

[0057] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / action>

Claims

1. A log anomaly detection method based on large model knowledge distillation, characterized in that, include: Step 1: Use the Drain algorithm to obtain the log template and use the sliding window method to split the continuous log messages into log sequences, and use a log sequence as a sample; Step 2: Guide DeepSeek-R1 to focus on log anomaly detection using designed prompts and samples; Step 3: Input the training set into DeepSeek-R1, guide it with the prompt words, and obtain the structured anomaly analysis report generated by DeepSeek-R1. Use the anomaly analysis report as a knowledge example for subsequent knowledge distillation, and enhance the knowledge example. The training set consists of multiple labeled samples; Step 4: Use DeepSeek-R1 as the teacher model and Qwen3-4B as the student model. Train Qwen3-4B using augmented knowledge examples and knowledge distillation. The training process is guided by a loss function that matches the learning objective. At the same time, use the LoRA method to fine-tune Qwen3-4B and use the fine-tuned Qwen3-4B to perform log anomaly detection.

2. The log anomaly detection method based on large model knowledge distillation according to claim 1, characterized in that, In step 2, the prompt words include a task description, a formatted statement, and human knowledge injection; the task description is used to inform DeepSeek-R1 that log anomaly detection is a binary classification problem and to explain the occurrence of the anomaly; the formatted statement is used to request the response to be output in JSON format; and the human knowledge injection is used to introduce labeled log examples.

3. The log anomaly detection method based on large model knowledge distillation according to claim 1, characterized in that, The training set was obtained by continuously sampling a public log dataset.

4. The log anomaly detection method based on large model knowledge distillation according to claim 1, characterized in that, The anomaly analysis report includes log sequence ID, anomaly summary, key events, potential impact, and handling recommendations.

5. The log anomaly detection method based on large model knowledge distillation according to claim 1, characterized in that, In step 3, enhancing the knowledge examples includes: Perform language validation on log anomaly detection descriptions, retaining only English descriptions; By leveraging the intermediate states and output structure features of DeepSeek-R1 in a single inference iteration, combined with a pre-defined domain rule base, the confidence score of knowledge examples is calculated, and the knowledge examples are filtered based on the confidence score.

6. The log anomaly detection method based on large model knowledge distillation according to claim 5, characterized in that, The confidence score of the knowledge example is calculated as follows: In the formula, in This represents the confidence score of knowledge example x. , and These are weighting coefficients. The score for matching the domain rules of x. This represents the DeepSeek-R1 internal confidence score for x. The structural integrity score for x is calculated, where n represents the number of key tokens. Represents the set of key tokens. Indicates that in the given preceding context Under the given conditions, the model predicts the probability of the i-th key token.

7. The log anomaly detection method based on large model knowledge distillation according to claim 1, characterized in that, In step 4, the loss function is: In the formula, in, It is the total loss function. It is a loss of structural consistency. It is semantic distillation loss. It is a task-oriented loss. These are weight parameters. For indicator functions, To analyze the failure penalty item, , K is a learnable parameter, representing the total number of predefined required fields. This indicates the k-th required field. This represents the output of the student model. These are the semantic vectors obtained by mean pooling from the last hidden state of the student model and the teacher model, respectively. It is the cosine similarity function. These are the probability distributions of the event → impact → suggestion causal chain within the student model and the teacher model, respectively. Denotes KL divergence, and It is an adjustable weighting coefficient. These represent the sets of domain terms extracted from the outputs of the student model and the teacher model, respectively. Used to calculate terminology consistency scores based on domain dictionary This represents the probability distribution of anomaly categories predicted by the student model for the current input log sequence. It is a high-confidence prediction of the teacher model validated by Log-Filter. It is an adjustable weighting coefficient. This is used to ensure that the student model's scoring and ranking on abnormal and normal samples are consistent with the teacher's. These represent the anomaly scores assigned to the input sequence by the student model and the teacher model, respectively.

8. The log anomaly detection method based on large model knowledge distillation according to claim 7, characterized in that, The fine-tuning of Qwen3-4B using the LoRA method includes: Load the pre-trained Qwen3-4B model weights as the base model, freeze all its original parameters, and then load the pre-trained weight matrix. The update amount of the pre-trained weight matrix is ​​constrained to a low-rank decomposition form: ,in , It is a predefined low-rank, significantly smaller than Dimensions and ; The knowledge examples enhanced in step 3 are used as training data. Each training sample consists of a log sequence and structured knowledge. The log sequence and structured knowledge are concatenated into an instruction fine-tuning format according to the prompt response template, which serves as the model input and expected output. Add a step-by-step instruction hierarchy to guide the model's analysis according to the process, requiring results to be output in JSON format. As the optimization objective, the AdamW optimizer is adopted, and gradient updates are performed in 4-bit quantization mode. During fine-tuning, only the matrix is ​​updated. and ,Keep Freeze the memory and accelerate computation using the Unsloth framework; Will Low-rank update volume with LoRA Add them together to generate an independent, lightweight log anomaly detection model.

9. A log anomaly detection system based on large model knowledge distillation, characterized in that, include: The preprocessing unit is used to obtain log templates using the Drain algorithm and to split continuous log messages into log sequences using the sliding window method, taking a log sequence as a sample. The domain knowledge injection unit is used to guide DeepSeek-R1 to focus on the domain of log anomaly detection through designed prompts and samples; The knowledge example generation unit is used to input the training set into DeepSeek-R1, guide it with the prompt words, obtain a structured anomaly analysis report generated by DeepSeek-R1, use the anomaly analysis report as a knowledge example for subsequent knowledge distillation, and enhance the knowledge example. The training set consists of multiple labeled samples; The model training and detection unit uses DeepSeek-R1 as the teacher model and Qwen3-4B as the student model. It trains Qwen3-4B using augmented knowledge examples and knowledge distillation. The training process is guided by a loss function that matches the learning objective. At the same time, the LoRA method is used to fine-tune Qwen3-4B, and the fine-tuned Qwen3-4B is used for log anomaly detection.

Images