Industrial log anomaly judgment method and system based on end-to-end large model
By employing an end-to-end large-scale industrial log anomaly detection method, and utilizing preprocessing and LoRA fine-tuning techniques, the problem of insufficient log detection accuracy and generalization ability is solved, achieving efficient and reliable log anomaly detection, applicable to scenarios such as cloud computing, distributed systems, and edge computing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN UNIV OF TECH
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-26
AI Technical Summary
Existing log anomaly detection methods have low detection accuracy and weak generalization ability when dealing with highly unstructured, semantically ambiguous, and context-complex log text.
An industrial log anomaly detection method based on an end-to-end large model is adopted. The method preprocesses the original industrial system log data, constructs a training set, and fine-tunes the large language model using a low-rank matrix LoRA. Combined with a grouped query attention mechanism and a gated neural network, the method outputs the anomaly category identification results.
It significantly improves detection accuracy and reliability, reduces resource consumption, and has good scalability and applicability, enabling its application in various industrial scenarios.
Smart Images
Figure CN122286476A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for identifying anomalies in industrial logs based on an end-to-end large model. Background Technology
[0002] With the continuous advancement of natural language processing (NLP) technology, large language models have emerged as a new research direction. Large language models possess massive parameters and powerful pre-training capabilities, enabling them to learn language patterns, domain knowledge, and complex reasoning abilities from vast amounts of corpora. However, despite their powerful capabilities in NLP and good adaptability across multiple tasks, large language models still face some key challenges when applied to log anomaly detection. Existing log anomaly detection methods have significant shortcomings in natural language understanding, especially when dealing with highly unstructured, semantically ambiguous, and context-rich log texts. Traditional methods struggle to effectively capture the semantic features behind abnormal behavior, resulting in low detection accuracy and weak generalization ability.
[0003] Therefore, there is an urgent need for an industrial log anomaly detection method and system based on an end-to-end large model to address the shortcomings of existing technologies. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for judging industrial log anomalies based on an end-to-end large model, so as to solve the technical problems mentioned in the background art.
[0005] To achieve the above objectives, the first aspect of this invention proposes a method for anomaly detection in industrial logs based on an end-to-end large model, comprising: Obtain raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data; The preprocessed industrial system log data is embedded into a preset prompt word template, and a training set is built based on the prompt word template; The training set is input into the pre-trained large language model, and the large language model is fine-tuned using the low-rank matrix LoRA to obtain the fine-tuned large language model. The industrial system log data to be detected is input into a trained large language model, which outputs the identification results of the anomaly category.
[0006] Furthermore, the specific method for preprocessing the raw industrial system log data is as follows: The raw industrial system log data is manually annotated with supervisory information, which includes normal labels and abnormal labels.
[0007] Furthermore, the prompt template includes detection instructions, input log content, and contextual examples.
[0008] Furthermore, the large language model specifically includes: The word segmentation layer is used to segment the input text using the word segmenter of the large language model and convert the input text into a token sequence. An embedding layer is used to map token sequences into high-dimensional semantic vectors; The feature extraction layer is used to convert high-dimensional semantic vectors into hidden state vectors; The output layer is used to map the hidden state vectors to a dimension with the same size as the vocabulary and use the Softmax function to generate a probability distribution of the output token sequence.
[0009] Furthermore, the feature extraction layer consists of multiple stacked Transformer decoders, each of which includes a grouped query attention mechanism and a feedforward neural network module, and is stably trained through residual connections and normalization layers. The grouped query attention mechanism includes a query projection layer, which maps the input high-dimensional semantic vector to a query matrix; a key projection layer, which maps the high-dimensional semantic vector to a key matrix; a value projection layer, which maps the high-dimensional semantic vector to a value matrix; and an output projection layer, which performs a linear transformation on the attention-weighted value vector to obtain the output features. The feedforward neural network employs a gating mechanism, including a first linear transformation layer and a second linear transformation layer; The low-rank matrix LoRA is inserted into the query matrix and value matrix in the grouped query attention mechanism and the first linear transformation layer in the feedforward neural network to achieve fine-tuning of the parameters of the large language model.
[0010] Furthermore, the specific method for fine-tuning a large language model using a low-rank matrix LoRA is as follows: The low-rank matrix LoRA includes the input projection matrix A and the output projection matrix B. When the large language model is fine-tuned, the input projection matrix A and the output projection matrix B are updated, and the original main parameters of the large language model are frozen. The LoRA update formula for low-rank matrices is used to update the low-rank matrix LoRA, thereby enabling fine-tuning of the large language model.
[0011] Furthermore, the LoRA update formula for a low-rank matrix is: ; in, Let A represent the projection matrix. For the newly added low-rank trainable matrix, This represents the original main parameter matrix of the large model.
[0012] A second aspect of this invention proposes an industrial log anomaly detection system based on an end-to-end large model, comprising: The acquisition module is used to acquire raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data. The module is used to embed preprocessed industrial system log data into a preset prompt word template and build a training set based on the prompt word template. The fine-tuning module is used to input the training set into the pre-trained large language model and fine-tune the large language model using the low-rank matrix LoRA to obtain the fine-tuned large language model. The detection module is used to input the industrial system log data to be detected into a trained large language model and output the identification results of the anomaly category.
[0013] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program, the computer program being stored in the memory and configured to be executed by the processor to implement the method.
[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the method.
[0015] The beneficial effects of the technical solutions provided by the embodiments of the present invention include: 1. Simplify the processing flow and reduce error propagation: This invention adopts an end-to-end modeling approach, which significantly reduces information loss and error accumulation between various links, and improves the stability and maintainability of the entire system.
[0016] 2. Effectively suppressing the "illusion" phenomenon and improving detection reliability: When large language models are applied to log scenarios, the "illusion" problem may occur, meaning that the results generated by the model seem reasonable but do not match the actual semantics. This invention, by designing structured task templates, adopting an instruction fine-tuning mechanism, and combining context consistency enhancement methods, successfully avoids erroneous reasoning in the generation process, greatly improving the reliability and credibility of anomaly detection.
[0017] 3. Reduce resource consumption and improve training efficiency: This invention adopts the LoRA low-rank matrix adaptation mechanism, which enables the model to maintain high performance while significantly reducing the consumption of computing resources during fine-tuning.
[0018] 4. Wide applicability and good scalability: The technical solution of this invention does not depend on a specific log format or system platform, and has good scalability and versatility. It can be widely applied to different industrial scenarios such as cloud computing platforms, distributed systems, and edge computing, and has strong engineering application value and market prospects. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating an embodiment of the present invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Example 1 The first aspect of this invention proposes a method for anomaly detection in industrial logs based on an end-to-end large model, such as... Figure 1 As shown, the method includes: Obtain raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data; Furthermore, the specific method for preprocessing the raw industrial system log data is as follows: The raw industrial system log data is manually annotated with supervisory information, which includes normal labels and abnormal labels.
[0022] The preprocessed industrial system log data is embedded into a preset prompt word template, and a training set is built based on the prompt word template; Furthermore, the prompt template includes detection instructions, input log content, and contextual examples.
[0023] In this embodiment, to ensure the large language model correctly parses the constructed prompt input, it is necessary to adhere to the model input format specifications. This involves adding elements such as [CLS] (start marker) and [SEP] (separator marker) to the input sequence. <inst>(Instruction marker) <input> (Input content marker) <output>(Expected output token) and other special token sequences.
[0024] Labeling helps the model distinguish different information segments and establish a clear input structure.
[0025] For example, the GPT class model adopts the following structure: " <inst>Please determine if the following logs are abnormal: <input> [Log Content] <output>" Set the mask and alignment strategy for the input sequence, unify the sequence length, and construct a dataset format for instruction-oriented fine-tuning to meet the requirements of batch training.
[0026] The training set is input into the pre-trained large language model, and the large language model is fine-tuned using the low-rank matrix LoRA to obtain the fine-tuned large language model. Furthermore, the large language model specifically includes: The word segmentation layer is used to segment the input text using the word segmenter of the large language model and convert the input text into a token sequence. Specifically, given a dataset of instructions One of the instructions The token sequence obtained from word segmentation .
[0027] An embedding layer is used to map token sequences into high-dimensional semantic vectors; Specifically, the embedding layer maps each token sequence in the corresponding vocabulary to a high-dimensional vector space through a lookup table to obtain a high-dimensional semantic vector. This process transforms discrete word segmentation into continuous vector representations for subsequent feature extraction layers.
[0028] The pre-trained large language model is trained on a massive corpus. Its large vocabulary improves the model's coverage of low-frequency words and proper nouns in the field of log anomaly detection, and can solve the problem of out-of-vocabulary words to a greater extent, thereby improving the model's adaptability to multi-source heterogeneous logs.
[0029] Define the embedding matrix ,in, For vocabulary list, For vocabulary size, Let be the dimension of the embedding vector. For the input token sequence... The embedding is represented as ,have The embedding vector for each token sequence is... .
[0030] The feature extraction layer is used to convert high-dimensional semantic vectors into hidden state vectors; Furthermore, the feature extraction layer includes stacked Transformer decoders, each of which includes a grouped query attention mechanism and a feedforward neural network; The grouped query attention mechanism includes a query projection layer, which maps the input high-dimensional semantic vector to a query matrix; a key projection layer, which maps the high-dimensional semantic vector to a key matrix; a value projection layer, which maps the high-dimensional semantic vector to a value matrix; and an output projection layer, which performs a linear transformation on the attention-weighted value vector to obtain the output features. The feedforward neural network employs a gating mechanism, including a first linear transformation layer and a second linear transformation layer; The low-rank matrix LoRA is inserted into the query matrix and value matrix in the grouped query attention mechanism and the first linear transformation layer in the feedforward neural network, respectively.
[0031] Specifically, the large language model employed in this invention utilizes a grouped query attention mechanism. Unlike traditional multi-head attention, where each head independently learns the linear transformations of the query, key, and value, grouped query attention divides all attention heads into several groups. Within the same group, the query transformation matrices of each attention head remain independent, while sharing the same key and value transformation matrices. Compared to multi-head attention, grouped query attention reduces the amount of key-value caching required, accelerates inference speed, and approximates the performance of multi-head attention.
[0032] Let the input be Initial input Query matrix Parameter matrix .in, Representing dimension, This represents the dimension of the key vector.
[0033] Query vector The calculation formula is: Key vector The calculation formula is: Value vector The calculation formula is: ; Large language models employ rotational positional encoding Process the query and key, and add position information to each token sequence: .
[0034] The large language model used in this invention is a causal language model. When calculating the scaled dot product attention, a mask matrix is needed to ensure that the model only focuses on the current and previous token sequences. To shield the parts that should not be included in the calculation: ; in, and These are the query and key matrices after rotation and position encoding processing, respectively.
[0035] Mask M sets the attention score corresponding to the current token sequence position to - before calculating Softmax. This results in the values of these positions being 0 after Softmax.
[0036] The feedforward neural network module following the attention module employs a gating mechanism, and its computation can be represented as: in, , , , This represents element-wise multiplication. This represents the activation function.
[0037] The output layer is used to map the hidden state vectors to a dimension with the same size as the vocabulary and use the Softmax function to generate a probability distribution of the output token sequence.
[0038] Specifically, the output layer maps the hidden state vectors to the vocabulary size ( The same dimensions are used, and the Softmax function is used to generate the probability distribution of each token sequence to implement the next token sequence. Prediction: ;in, .
[0039] Furthermore, the specific method for fine-tuning a large language model using a low-rank matrix LoRA is as follows: The low-rank matrix LoRA includes the input projection matrix A and the output projection matrix B. When the large language model is fine-tuned, the input projection matrix A and the output projection matrix B are updated, and the original main parameters of the large language model are frozen. The LoRA update formula for low-rank matrices is used to update the low-rank matrix LoRA, thereby enabling fine-tuning of the large language model.
[0040] Furthermore, the LoRA update formula for a low-rank matrix is: ; in, Let A represent the projection matrix. For the newly added low-rank trainable matrix, This represents the original main parameter matrix of the large model.
[0041] The industrial system log data to be detected is input into a trained large language model, which outputs the identification results of the anomaly category.
[0042] To verify the technical effectiveness of this invention, the following comparative experiment was designed: Experimental environment: Linux server (NVIDIA A100-PCIE-40GB GPU), Python 3.10; Dataset: BGL dataset (BlueGene / L supercomputer logs, approximately 4.74 million records, including anomaly labels annotated by the administrator); Data partitioning: training / test set ratio 8:2, observation log sliding window size 10, step size 10; The basic model used in the experiment, the total number of model parameters, the number of LoRA parameters, and the trainable ratio are shown in Table 1: Table 1. Base Model Size and LoRA Parameter Ratio Configuration parameters: Optimizer: AdamW, initial learning rate 5×10⁻⁴ - 45×10⁻⁴ -4 ; Learning strategy: cosine annealing, warmup ratio 0.01, weight decay 0.1; LoRA fine-tuning parameters: r=8, α=16.
[0043] Batch and precision: Batch size is 32, and BF16 mixed precision training is used to accelerate convergence.
[0044] The experimental results are shown in Table 2: Table 2. Impact of LoRA fine-tuning on log anomaly detection performance According to Table 2, the advantages of LogX are: 1. End-to-end advantages: LogX skips log parsing and directly maps the original log semantics, solving the problem of cascading error accumulation, and its F1 score surpasses that of traditional models. 2. Semantic Illusion Suppression: Supervised fine-tuning enables LLM to internalize industry-specific semantics, solving the illusion problem and ensuring reliability in industrial scenarios; 3. High efficiency and low power consumption: LoRA makes trainable parameters only account for 0.26% of the total, realizing the feasibility of industrial control log awareness under limited resources.
[0045] Example 2 A second aspect of this invention proposes an industrial log anomaly detection system based on an end-to-end large model, comprising: The acquisition module is used to acquire raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data. The module is used to embed preprocessed industrial system log data into a preset prompt word template and build a training set based on the prompt word template. The fine-tuning module is used to input the training set into the pre-trained large language model and fine-tune the large language model using the low-rank matrix LoRA to obtain the fine-tuned large language model. The detection module is used to input the industrial system log data to be detected into a trained large language model and output the identification results of the anomaly category.
[0046] Example 3 A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program, the computer program being stored in the memory and configured to be executed by the processor to implement the method described in Embodiment 1.
[0047] Example 4 A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the method described in Embodiment 1.
[0048] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0049] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0050] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0051] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.< / output> < / inst> < / output> < / inst>
Claims
1. A method for anomaly detection in industrial logs based on an end-to-end large model, characterized in that, include: Obtain raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data; The preprocessed industrial system log data is embedded into a preset prompt word template, and a training set is built based on the prompt word template; The training set is input into the pre-trained large language model, and the large language model is fine-tuned using the low-rank matrix LoRA to obtain the fine-tuned large language model. The industrial system log data to be detected is input into a trained large language model, which outputs the identification results of the anomaly category.
2. The specific method for preprocessing the original industrial system log data according to the end-to-end large model-based industrial log anomaly detection method described in claim 1 is as follows: The raw industrial system log data is manually annotated with supervisory information, which includes normal labels and abnormal labels.
3. The industrial log anomaly detection method based on an end-to-end large model according to claim 1, characterized in that, The prompt template includes detection instructions, input log content, and context examples.
4. The industrial log anomaly detection method based on an end-to-end large model according to claim 1, characterized in that, The large language model specifically includes: The word segmentation layer is used to segment the input text using the word segmenter of the large language model and convert the input text into a token sequence. An embedding layer is used to map token sequences into high-dimensional semantic vectors; The feature extraction layer is used to convert high-dimensional semantic vectors into hidden state vectors; The output layer is used to map the hidden state vectors to a dimension with the same size as the vocabulary and use the Softmax function to generate a probability distribution of the output token sequence.
5. The industrial log anomaly detection method based on an end-to-end large model according to claim 4, characterized in that, The feature extraction layer consists of multiple stacked Transformer decoders, each of which includes a grouped query attention mechanism and a feedforward neural network module, and is stably trained through residual connections and normalization layers. The grouped query attention mechanism includes a query projection layer, which maps the input high-dimensional semantic vector to a query matrix; a key projection layer, which maps the high-dimensional semantic vector to a key matrix; a value projection layer, which maps the high-dimensional semantic vector to a value matrix; and an output projection layer, which performs a linear transformation on the attention-weighted value vector to obtain the output features. The feedforward neural network employs a gating mechanism, including a first linear transformation layer and a second linear transformation layer; The low-rank matrix LoRA is inserted into the query matrix and value matrix in the grouped query attention mechanism and the first linear transformation layer in the feedforward neural network to achieve fine-tuning of the parameters of the large language model.
6. The industrial log anomaly detection method based on an end-to-end large model according to claim 1, characterized in that, The specific method for fine-tuning a large language model using low-rank matrix LoRA is as follows: The low-rank matrix LoRA includes the input projection matrix A and the output projection matrix B. When the large language model is fine-tuned, the input projection matrix A and the output projection matrix B are updated, and the original main parameters of the large language model are frozen. The LoRA update formula for low-rank matrices is used to update the low-rank matrix LoRA, thereby enabling fine-tuning of the large language model.
7. The industrial log anomaly judgment method based on an end-to-end large model according to claim 6, characterized in that, The LoRA update formula for a low-rank matrix is: ； in, Let A represent the projection matrix. For the newly added low-rank trainable matrix, This represents the original main parameter matrix of the large model.
8. An industrial log anomaly detection system based on an end-to-end large model, characterized in that, include: The acquisition module is used to acquire raw industrial system log data, preprocess the raw industrial system log data, and obtain preprocessed industrial system log data. The module is used to embed preprocessed industrial system log data into a preset prompt word template and build a training set based on the prompt word template. The fine-tuning module is used to input the training set into the pre-trained large language model and fine-tune the large language model using the low-rank matrix LoRA to obtain the fine-tuned large language model. The detection module is used to input the industrial system log data to be detected into a trained large language model and output the identification results of the anomaly category.
9. An electronic device, characterized in that, include: A memory, a processor, and a computer program, the computer program being stored in the memory and configured to be executed by the processor to implement the method of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of claims 1-7.