Abnormality detection method based on fast layer normalization attention SLNA mechanism

Abstract

An anomaly detection method based on a fast layer normalization attention SLNA mechanism comprises a log template extraction step and a log anomaly detection model construction implementation step, the log template extraction step is based on a cue word project and a large language model, redundant log template information is avoided, and the log anomaly detection model construction implementation step is based on the cue word project and the large language model. Therefore, variables and constants in the log messages are correctly distinguished, and the log messages with the same template are clustered; according to the log anomaly detection model construction implementation step, an anomaly detection encoder module is adopted, an Adapter adapter module is added, and partial fine adjustment is performed on the adapter module, so that the generalization ability of the log anomaly detection model is improved. According to the method, the utilization rate of the log anomaly detection model on the existing knowledge is enhanced, and the generalization ability of the log anomaly detection model is improved; by improving the multi-head attention mechanism, the anomaly detection speed is increased.

Description

Technical Field

[0001] The invention discloses a detection method, and in particular relates to an anomaly detection method and a device thereof based on a fast layer normalized attention (SLNA) mechanism. Background Art

[0002] In view of the current problems of many devices connected to the power Internet, blurred network security boundaries, and increasing security risks from network attacks, it is urgent to improve the level of network attack behavior detection technology to ensure the safe and stable operation of network edge devices. Device behavior will be recorded through logs, and the recorded log text information will be analyzed to evaluate the security status of the device. Therefore, by building a log-based intrusion detection system, the network security situation awareness and disposal level of edge devices can be improved, and the risk of power Internet being invaded and damaged from the external network can be reduced.

[0003] Traditional log anomaly detection usually first extracts templates from logs based on the tree model Drain algorithm [1]. However, this solution contains a large amount of redundant template information. For example, it will add redundant templates such as "BLOCK*ask<*>to replicate<*>to datanode(s)<*>" to the template "BLOCK*ask<*>to replicate<*>to datanode(s)<*>", which reduces the accuracy of the anomaly detection model. Reference [2] proposes to use prompt-based small sample learning to capture the pattern of templates. This method adaptively extracts a few most frequent variable tokens from the log to generate virtual labels for model training. However, for diverse and complex data sets, there may be no logs or variables that are representative of the entire data set because the data set contains large log and variable variances and unbalanced template distribution. This training process may limit the model to specific training data, resulting in poor generalization performance of the entire data set. Reference [3] parses log templates based on word frequency statistics. The algorithm first counts the frequency of all n-grams in the log (n = 1, 2, ... k, k is a setting parameter). Then, during parsing, first obtain the k-gram frequency of the log, select the k-gram with a frequency less than the k-gram threshold as a parameter candidate, then decompose the selected k-gram into k-1-gram, and then select the parameter candidate of k-1-gram from it until 2-gram. For any word in the log, if all its 2-grams are in the 2-gram parameter candidates, then the word is a parameter, otherwise it is a constant. Fix the constant and convert the parameter into a special symbol <*> to obtain the log template. However, in real scenarios, some log templates appear less frequently, and some log templates appear very frequently, which means that in those logs with less occurrences, the frequency of constants will also be very low, so the accuracy of log template parsing will be greatly reduced. Existing log template extraction methods are difficult to correctly distinguish between variables and constants in log messages, and can only roughly cluster log messages with the same template. This leads to redundant log events in the process of log anomaly analysis, which reduces the detection accuracy of the anomaly detection model. Therefore, the present invention provides a log template extraction technology based on a large language model, which can limit the output range through pre-set prompt words and avoid generating redundant log template information.

[0004] In addition, anomaly detection models often use LSTM, Transformer and other models to analyze log sequences. However, such models have high computational complexity and high requirements for device hardware, making it difficult to meet the requirements of real-time anomaly detection. For example, the Log Anomaly[6] model based on the LSTM model takes 20 minutes to detect whether a set of sequences has abnormal behavior. Reference [4] addresses the problem that the model effect deteriorates due to the evolution of log statements over time, performs word frequency statistics and word vector embedding on the input features, converts text information into semantic feature vectors, and constructs a bidirectional LSTM model for anomaly detection. Reference [5] proposes a probabilistic label estimation algorithm for unlabeled log sequences in the training set. The algorithm estimates the probability label of normal or abnormal for the sequence through cluster analysis and probability calculation. The GRU neural network with attention mechanism is used to train the anomaly detection model using the semantic vectors and probabilistic labels obtained by the word embedding model and the label estimation algorithm. Although the above method can accurately analyze log text information and detect abnormal behavior, the detection time is long and it is difficult to meet the real-time requirements of anomaly detection.

[0005] Deep learning is currently the most popular method in log anomaly detection, and Transformer is the most widely used model in this field. However, in the traditional Transformer model, the self-attention mechanism has a quadratic time complexity, that is, when the length of the input sequence is N, the amount of calculation is O(N 2 ), which will lead to very large computational and memory overhead when processing long sequences such as logs. Therefore, this patent provides a fast layer normalization attention mechanism by optimizing the self-attention mechanism, which greatly improves the efficiency of log anomaly detection.

[0006] In addition, due to the rapid development of log systems, the scale of logs continues to expand, and new types of logs are constantly added. The existing technology mainly trains models based on a specific log data set, such as BGL and HDFS, which results in the model's poor ability to detect newly emerging logs.

[0007] References:

[0008] [1]HEP, ZHU J, ZHENG Z, et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree; proceedings of the 2017 IEEE International Conference on WebServices(ICWS), F25-30June 2017, 2017[C].

[0009] [2]LE V H,ZHANG H.Log Parsing with Prompt-based Few-shot Learning;proceedings of the 2023 IEEE / ACM 45th International Conference on SoftwareEngineering(ICSE),F 14-20May 2023,2023[C].

[0010] [3]DAI H,LI H,CHEN C S,et al.Logram:Efficient Log Parsing Using n-Gram Dictionaries[J].IEEE Transactions on Software Engineering,2022,48(3):879-92.

[0011] [4]ZHANG X,XU Y,LIN Q,et al.Robust log-based anomaly detection onunstable log data[Z].Proceedings of the 2019 27th ACM Joint Meeting onEuropean Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering.Tallinn,Estonia;Association for ComputingMachinery.2019:807-17.10.1145 / 3338906.3338931

[0012] [5]YANG L,CHEN J,WANG Z,et al.Semi-Supervised Log-Based AnomalyDetection via Probabilistic Label Estimation;proceedings of the 2021 IEEE / ACM43rd International Conference on Software Engineering (ICSE),F 22-30May2021,2021[C].

[0013] [6]MENG W, LIU Y, ZHU Y, et al.Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs[Z].Proceedings of the 28th International Joint Conference on Artificial Intelligence.Macao, China; AAAI Press.2019: 4739-45 Summary of the invention

[0014] In order to solve the defects in the prior art, the present invention discloses an anomaly detection method based on a Swift Layer Normalization of Attention (SLNA) mechanism, and its technical solution is as follows:

[0015] An anomaly detection method based on a fast layer normalized attention (SLNA) mechanism is provided. The anomaly detection method includes log template extraction and log anomaly detection model construction and implementation steps, and is characterized by:

[0016] The log template extraction step is based on prompt word engineering and a large language model to avoid generating redundant log template information, thereby correctly distinguishing variables and constants in log messages and clustering log messages with the same template;

[0017] The log anomaly detection model construction implementation step improves the generalization ability of the log anomaly detection model by adopting an anomaly detection encoder module and adding an adapter module, and partially fine-tuning the Adapter module.

[0018] The present invention also discloses an anomaly detection device based on a fast layer normalized attention (SLNA) mechanism.

[0019] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0020] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes an electronic device, characterized in that it includes a processor and a memory; the memory stores computer-readable instructions, and the processor is used to run the computer-readable instructions, wherein the computer-readable instructions execute the above method when they are run.

[0021] Beneficial Effects

[0022] (1) The technology of the present invention is based on prompt word engineering and large language model (LLM) to accurately extract log templates and avoid generating redundant log information.

[0023] (2) The technology of the present invention constructs a log anomaly detection model and performs anomaly analysis through an anomaly detection encoder module, thereby improving the efficiency of log anomaly detection.

[0024] (3) The technology of the present invention introduces an adapter module, which improves the generalization ability of the log anomaly detection model, can adapt to the changes of logs over time and reduce the cost required for model updating. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of an anomaly detection method based on the fast layer normalized attention SLNA mechanism of the present invention;

[0026] Figure 2 This is a schematic diagram of an abnormality detection encoder module of the present invention;

[0027] Figure 3 This is a schematic diagram of the adapter module;

[0028] Figure 4 This is a schematic diagram of extracting the content of the log template task prompt and the question and answer results of the present invention. DETAILED DESCRIPTION

[0029] The anomaly detection method based on the fast layer normalized attention SLNA mechanism of the present invention needs to go through two stages: log template extraction and anomaly analysis. The operation process of the method is as follows: Figure 1 shown.

[0030] First, extract the log template. If you want to detect log anomalies, you must first distinguish constants and variables from the original log information and organize them into corresponding templates.

[0031] This method is based on prompt word engineering and large language models to accurately extract log templates and reduce the generation of redundant log information. Large language models (LLMs) can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. by training models on large amounts of text data. LLMs are characterized by their large scale and billions of parameters, which help them learn complex patterns in language data. Prompt word engineering is an artificial intelligence technology that improves the performance of AI by designing and improving the question-answering method with AI. The goal of prompt word engineering is to create highly effective and controllable AI systems that can perform specific tasks accurately and reliably. For each log text of the log template to be extracted, based on the pre-defined prompt word template, similar logs are selected from the candidate log sample set, combined with the corresponding pre-labeled log template information, the corresponding prompt words are generated, and interactive questions are asked to the large language model model to obtain the corresponding log template. The specific steps are as follows:

[0032] Step 1 Use regular expressions to structure the logs and remove irrelevant information and operations: For each application log, construct a corresponding regular expression to convert the unstructured log text into a structured log. For example, for the HDFS service log, construct a regular expression “^(?P <date>.*?)\\s+(?P <time>.*?)\\s+(?P <pid>.*?)\\s+(?P <level>.*?)\\s+(?P <component>.*?):\\s+(?P <content>.*? )$", which is used to extract the date, time, process ID, log level, HDFS service component information, and log content from the log. The parsed useless information, such as date, time, and other redundant information, is discarded, and the key information is retained.

[0033] Step 2: First, the log text items are segmented into tokens by using the BERT word embedding model (Bidirectional Encoder Representations from Transformers, abbreviated as BERT, a bidirectional pre-trained language model based on Transformer that can learn the deep semantics of text). Each token corresponds to an embedding vector. Then, using the self-attention mechanism, the BERT word embedding model can generate a log semantic feature vector.

[0034] After obtaining these vectors, we need to apply the cosine similarity formula to calculate the similarity between them. The formula definition is shown in formula (1):

[0035]

[0036] Where i and j represent the subscripts of different semantic vectors, respectively. i and v j Represents the two log semantic feature vectors whose similarity is to be calculated, represents the dot product of these two vectors, ||v i || means calculating vector v i The second norm of ||v i ||×||v j || represents the product of the second norms of two vectors.

[0037] The cosine similarity formula determines the angle between the vectors by calculating the dot product of the two vectors and dividing it by the product of their moduli, resulting in a value between -1 and 1. The closer this value is to 1, the more similar the two logs are; the closer it is to -1, the less similar they are. In this way, the log similarity can be calculated in the end.

[0038] Step 3 is based on the DPP (Determinantal Point Process) algorithm and the calculated log similarity. The candidate log sample set is selected from the vectorized log template matrix to maximize the log diversity and is manually labeled. This reduces labor costs and the potential risk of inductive bias caused by the imbalanced distribution of log sample templates. The DPP algorithm selects N log semantic feature vectors from the existing set. Select candidate log sample set Through formula (2) and By enumerating each subset in the , we can get the corresponding candidate log sample set

[0039]

[0040] in represents the selected candidate log sample set; represents the number of candidate log sample sets, Indicates that the size of the candidate log sample set is k; Represents the combination of each log semantic feature vector in the selected candidate log sample set, that is, Represents the semantic feature vector corresponding to a log in the candidate log sample set; express The transpose of a matrix; Representation calculation The value of the determinant of the matrix multiplication result; Function means when the expression When the maximum value is taken, the corresponding parameter The value of .

[0041] However, the computational complexity of enumeration is too high, so we improve formula (2). Based on the log semantic feature similarity calculated in step 2, we construct a symmetric similarity matrix L i，j = cos(v i , v j ), where L i，j Represents each log semantic feature vector v i v j The similarity between them, and at the beginning of the algorithm, set the set is an empty set. Select the first log i that is least similar to other logs * , usually choose the element that maximizes the determinant as shown in formula (3):

[0042]

[0043] where i * represents the initial log selected; Function means when the expression L i，i When the maximum value is taken, it corresponds to the value of log i. After that, each round continuously finds logs that satisfy formula (4) from the remaining unselected logs and adds them to the set In, until the collection The size of is equal to the size k of the set candidate log sample set:

[0044]

[0045] in represents the logs that are not selected into the candidate log sample set among all the logs; j represents the logs that are selected to be added to the candidate log sample set in this round; It represents the similarity matrix after adding j to the candidate log sample set; Indicates the size of the determinant of the corresponding similarity matrix. Through the DPP algorithm, log information can be selected with maximum diversity to avoid the problem of inaccurate parsing results during template parsing due to uneven log distribution. Step 4: Based on the KNN algorithm (K-Nearest Neighbors, referred to as KNN, a basic classification and regression algorithm), select log samples similar to the log to be parsed from the candidate log sample set, and sort them from low to high according to the similarity to construct prompt words. The content of the log template parsing prompt word is as follows Figure 4 It consists of three parts. First, through the question-answering structure statement, the large language model can clearly understand the requirements of this question-answering task, such as Figure 4 As shown, this prompt word limits the large language model to only " <prompt> ”"< / prompt> "The logs between the corresponding log templates are extracted and the results are converted to " <extraction> ： <start>result <end>" format output, which is convenient for later use of regular expressions" <start> (.+) <end>"Extract the corresponding log template. Then, use the KNN algorithm to select the log sample and the corresponding log template that are most similar to the log to be parsed from the candidate sample data set, and generate examples based on the question-answering task requirements specified in the previous step for contextual learning by the large language model. Finally, put the log to be parsed this time into" <prompt> ”"< / prompt> " and merge all the above contents to finally construct the prompt word corresponding to the log template parsing. This step can utilize the prior knowledge and generalization ability of the large language model pre-trained, without the need to repeatedly train the log template parsing models corresponding to different services, thus reducing the model training cost. In addition, since the prompt words are selected from the most similar log samples from the candidate log set with the maximum log diversity, the large language model can more accurately understand the different log expressions under the same template, thereby reducing the generation of redundant log templates.

[0046] Step 5: Input the constructed log template parsing prompt words into the Large Language Model (LLM) (an artificial intelligence model trained with deep learning algorithms that can understand and generate natural language, and has the ability to process complex language structures, understand context, and generate coherent text). Through its internal neural network structure, especially the trained word embedding and context encoder, the log template parsing prompt words are identified. The large language model will associate the log template parsing prompt words with specific log templates based on the patterns and association rules learned during training, thereby extracting the corresponding log template.

[0047] After that, the log sequence extracted from the log template is analyzed for anomalies. In order to improve the efficiency of log anomaly detection, the following Figure 2 Anomaly detection encoder module.

[0048] The final output of the anomaly detection encoder module is:

[0049]

[0050] where x is the log semantic feature vector input to the model, is the output of the anomaly detection encoder module, d represents the dimension of the semantic feature vector. Q and K represent the query matrix and key matrix extracted after the semantic feature vector x is linearly transformed. Norm represents layer normalization, Linear represents linear transformation, represents the self-learning weight parameters of the matrix Q, represents matrix multiplication, ⊙ represents the multiplication of elements at corresponding positions of the matrix, ∑ dim＝0 (·) represents the result of adding the matrix (·) row by row.

[0051] The loss function for log anomaly detection model training is defined as follows:

[0052]

[0053] where x i and i Represents the training data set The input and actual abnormal results, N represents the size of the sample data, Θ f (x i ) to represent the output of our model, Loss a Represents the loss function of the log anomaly detection model.

[0054] In addition, in order to adapt to the changes in logs over time and reduce the cost of updating the log anomaly detection model, an Adapter module is added to the log anomaly detection model, and the Adapter module is partially fine-tuned to improve the generalization ability of the log anomaly detection model. The specific steps are as follows:

[0055] Step 1: First, determine whether the log anomaly detection model needs to be updated based on manual feedback. After that, it is necessary to collect the error data handled by security operators every day, which includes but is not limited to log sequences that are mistakenly identified as security threats. Finally, these false positive log sequences are stored in a central knowledge base for further analysis and processing.

[0056] Step 2: First, set a false alarm rate threshold. Once the actual false alarm rate exceeds this threshold, the security operator can take action. They will use the previously collected false alarm data to optimize the parameters of the log anomaly detection model. This process may include adjusting the model's algorithm, adding new training data, or improving the model's feature extraction mechanism. Through these optimizations, the model can better understand which are real security threats and which are false alarms, thereby improving the accuracy of the model.

[0057] Step 3: In order to further enhance the robustness of the log anomaly detection model, an Adapter module is added to the model, such as Figure 3 As shown. The Adapter module is a key component of the model, which helps the model better adapt to new data and environmental changes. By fine-tuning the parameters of the Adapter module, you can ensure that the model can make full use of what it has learned while enhancing its generalization ability to new situations. This fine-tuning can be automatic or manual, depending on the complexity of the model and the operator's expertise.

[0058] In summary, the present invention first reduces redundant log templates based on prompt word engineering and large-scale language model technology, and strengthens the generalization ability of template parsing. Secondly, based on the fast layer normalized attention mechanism, a log anomaly detection model is constructed to improve the efficiency of log anomaly detection. Finally, based on the Adapter adapter module, the log anomaly detection model is partially fine-tuned to improve the generalization ability of the log anomaly detection model and reduce the cost of model retraining. Through the above content, the shortcomings of the existing log anomaly detection technology are improved, providing a more solid guarantee for the safety and stability of the power system, and promoting the transformation and upgrading of the power industry towards digitalization, intelligence, and greening.

[0059] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.< / end> < / start> < / end> < / start> < / extraction> < / content> < / component> < / level> < / pid> < / time> < / date>

Claims

1. An anomaly detection method based on the fast layer normalized attention (SLNA) mechanism, the anomaly detection method includes log template extraction and log anomaly detection model construction implementation steps, and is characterized by: The log template extraction step is based on prompt word engineering and a large language model to avoid generating redundant log template information, thereby correctly distinguishing variables and constants in log messages and clustering log messages with the same template; The log anomaly detection model construction implementation step is to add an Adapter module to the anomaly detection encoder module, and partially fine-tune the Adapter module to improve the generalization ability of the log anomaly detection model.

2. The anomaly detection method based on the fast layer normalized attention (SLNA) mechanism according to claim 1, characterized in that: The log template extraction step includes the following contents: Step 1: Use regular expressions to structure the logs to remove irrelevant information and operations; Step 2: Use the BERT word embedding model to convert log text items into semantic vectors, and calculate the cosine similarity between these vectors as shown in formula (1): Where: i, j represent the subscripts of different semantic vectors, v i and v j Represents the two log semantic feature vectors whose similarity is to be calculated, represents the dot product of these two vectors, ||v i || means calculating vector v i The second norm of ||v i ||×||v j || represents the product of the second norm of two vectors; Step 3: Based on the determinant point process DPP algorithm and the calculated log similarity, the candidate log sample set is selected from the vectorized log semantic feature matrix to maximize the log diversity and manually annotate it. The calculation formula is shown in formula (2): in: represents the selected candidate log sample set; represents the number of candidate log sample sets, V represents the size of the candidate log sample set as k; S Represents the combination of each log semantic feature vector in the selected candidate log sample set, that is, Represents the semantic feature vector corresponding to a log in the candidate log sample set; Indicates V S The transpose of a matrix; Representation calculation The value of the determinant of the matrix multiplication result; Function means when the expression When the maximum value is taken, the corresponding parameter The value of Step 4: Based on the KNN algorithm, select log samples similar to the log to be parsed from the candidate log sample set, sort them from low to high according to the similarity, and construct prompt words; Step 5 inputs the prompt word into the large language model LLM to extract the corresponding log template.

3. The anomaly detection method based on the fast layer normalized attention (SLNA) mechanism according to claim 1, characterized in that: The log anomaly detection model construction implementation steps include the following steps: Step 1: First, determine whether the log anomaly detection model needs to be updated based on manual feedback: collect error data processed by security operators every day, and store the log sequences identified as false positives in the knowledge base; Step 2: When the false alarm rate reaches the specified threshold, the security operator can use the collected false alarm data to optimize the log anomaly detection model parameters, so that the log anomaly detection model can be updated for related services; Step 3: In order to enhance the robustness of the log anomaly detection model, an Adapter module is added to the log anomaly detection model. By fine-tuning the parameters of the Adapter module, the generalization ability of the log anomaly detection model is enhanced.

4. An anomaly detection device based on the fast layer normalized attention (SLNA) mechanism, the anomaly detection device comprising a log template extraction module and a log anomaly detection model construction implementation module, characterized by: The log template extraction module is based on prompt word engineering and a large language model to avoid generating redundant log template information, thereby correctly distinguishing variables and constants in log messages and clustering log messages with the same template; The log anomaly detection model construction implementation module adds an Adapter module to the anomaly detection encoder module and partially fine-tunes the Adapter module to improve the generalization ability of the log anomaly detection model.

5. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute the method according to any one of claims 1 to 3 when the program is executed.

6. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 1 to 3 when executed.

Images