Automatic log anomaly detection method and system based on deep learning

Abstract

The invention relates to an automatic log anomaly detection method and system based on deep learning. The method comprises the following steps: extracting a representative log template sequence; encoding the log template sequence to generate a high-dimensional semantic vector for expressing the log template; a CNN-BiLSTM-Attention model is constructed, and the difference between the predicted value and the actual value of each data point is output based on the high-dimensional semantic vector; adopting a supervised learning method to train and optimize the model; reasoning to-be-tested log data through the trained model, and if the difference between a predicted value and an actual value exceeds a set threshold value, triggering an alarm. According to the method, the log template extraction capability based on the large language model, the semantic extraction capability of the Sension-BERT and the log sequence feature capture advantage of the CNN-BiLSTM-Attention model are fully played, and the accuracy and efficiency of log anomaly detection are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of Internet, and specifically relates to an automatic log anomaly detection method and system based on deep learning. Background Art

[0002] With the rapid development of information technology, the complexity of enterprise-level application systems and services has gradually increased. These systems are usually composed of multiple microservices and components. In order to ensure the high availability and stability of the system, logging has become one of the key monitoring methods. Logs not only record the operating status and various events of the system, but also provide an important basis for system anomaly detection, troubleshooting and performance optimization. However, as the scale of the system expands, the volume of log data has also increased exponentially. How to quickly and accurately detect anomalies from massive logs has become an important challenge currently faced.

[0003] There have been many studies on log-based anomaly detection methods. As the scale and complexity of existing systems continue to increase, it has become infeasible to manually check logs to find anomalies. Therefore, the need for automated anomaly detection for log information is becoming increasingly urgent. Traditional rule-based and statistical anomaly detection methods are difficult to adapt to complex and changing system environments due to their reliance on prior knowledge and fixed thresholds. This has led to widespread attention to log anomaly detection methods based on machine learning and deep learning.

[0004] Log anomaly detection based on machine learning uses the characteristics of log data to identify normal and abnormal patterns by training models. For example, common methods include using support vector machines (SVM), random forests, and k-nearest neighbor algorithms (KNN). These methods can detect anomalies in new data by learning patterns in historical log data. However, traditional machine learning methods usually rely on feature engineering, require expert knowledge to extract effective features, and may face performance bottlenecks when processing large-scale log data.

[0005] In contrast, deep learning-based log anomaly detection can automatically extract features from data, reducing reliance on feature engineering, and performs well when processing large-scale data and complex patterns. In recent years, researchers have proposed a variety of deep learning-based log anomaly detection methods, such as using convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM). Using these technologies to extract log sequence feature information for log anomaly detection has a certain degree of robustness, but there is still room for improvement in the accuracy of anomaly detection, so a deep learning model that is more suitable for capturing log sequence features is still needed. Summary of the invention

[0006] To solve the above problems, the present invention proposes an automated log anomaly detection method and system based on deep learning, which improves the accuracy of log anomaly detection.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows: An automated log anomaly detection method based on deep learning includes the following steps: Input the original log data into the log parsing module based on the large language model to extract representative log template sequences; Use the Sentence-BERT model to encode the log template sequence and generate a high-dimensional semantic vector expressing the log template; A CNN-BiLSTM-Attention model is constructed, and a supervised learning method is adopted to train and optimize the CNN-BiLSTM-Attention model using normal and abnormal log data with historical annotations; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different moments in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The trained CNN-BiLSTM-Attention model is used to infer the log data to be tested. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

[0008] Furthermore, the log parsing module based on the large language model extracts a log template set, including: For a set of raw log data input, the log messages are preprocessed through regular expressions to remove some variable contents in the logs, such as IP addresses or URLs, so as to prepare for subsequent log parsing operations.

[0009] After preprocessing, a set of log data is obtained. The length of each log is calculated for this set of log data, and these different length values ​​are formed into a unique length set. The length of the log refers to the number of characters or words in this text.

[0010] Create a corresponding number of buckets based on the number of unique length sets. Each bucket is set up to store log messages of different lengths. Then distribute the logs to different buckets based on their lengths.

[0011] For log messages in the bucket, calculate the information entropy for each log : , Refers to a symbol in the log, including words or characters. is the probability of this symbol appearing in the log.

[0012] After calculating the information entropy value of each log, the logs are sorted from high to low according to these information entropy values. The logs with high entropy values ​​are placed at the front, and the logs with low entropy values ​​are placed at the back.

[0013] The sorted logs are divided into multiple layers (for example, n layers), and then logs are selected from these layers one by one and added to the sample set S as the center of subsequent log clustering.

[0014] For each layer of logs, each log is checked in turn. The first word or character of each log is recorded as T(x).

[0015] For the first symbol T(x) of the current log, it has not been selected before (ie: T(x) ST(x)), then select this log and add it to the sample set S. Here ST represents the set of words or symbols that have been selected.

[0016] If some words or symbols in the current log have been selected for a predefined number of times, this log is selected and added to the sample set S.

[0017] This process will continue until the size of the selected sample set S reaches the predetermined number k, and the log selection process will stop.

[0018] For the selected log sample set, the Jaccard similarity method is used to calculate the similarity between these log samples. If the similarity is higher than a certain threshold, one of the log samples is deleted.

[0019] Select several log samples from the log message layer of each Bucket as log clustering centers , for each log message And each cluster center Calculating distance : , Indicates log messages Middle The symbol of the position; Represents the cluster center Middle The symbol of the position; Indicates that when , the value is 1, otherwise it is 0, according to each log message With each cluster center Distance , each log message Assign to the closest cluster center.

[0020] The log message of each cluster center is input into the large language model, and a context learning method is used to parse the log message and generate a log template. Context learning means that some sample log templates are used as context learning objects of the large language model. The cluster centers of the sample log template and the log template to be generated are The Prompt is input into the large language model, which automatically learns which parts of the log message to be parsed are fixed (i.e., part of the log template) and which parts are variables (i.e., log template < > part) to generate a log template and store it in the log template library.

[0021] For newly input log messages that do not match a log template in the log template library, the KNN algorithm is used to calculate the similarity between the log message to be parsed and the log messages in the log sample set and the generated log templates, and the most similar h log templates and log messages are selected to form a prompt input into the large language model, and the log template output by the large model is updated to the log template library. The prompt format example is as follows: Please extract the log template from the log message to be parsed according to the log message M, log template T, and log template generation principle P in these examples; example log templates T1, T2..., example log messages: M1, M2..., log template extraction principles: P1, P2..., log messages to be parsed: L1, L2...

[0022] Furthermore, the use of the Sentence-BERT model to encode the log template sequence to generate a high-dimensional semantic vector expressing the log template includes: The Sentence-BERT model receives a series of log templates as input. When processing the log templates, each log template can be regarded as a sentence.

[0023] These input sentences are encoded by the BERT model. The BERT model is a deep bidirectional Transformer that can understand the contextual information in the sentence. During the encoding process, the BERT model converts each symbol in the sentence into a vector.

[0024] After BERT encoding, the Sentence-BERT model aggregates the vectors of each symbol to generate a high-dimensional semantic vector representing the entire sentence.

[0025] Furthermore, the CNN layer receives a high-dimensional semantic vector expressing the log template, extracts local features of the log template sequence, and optimizes the feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence, including an input high-dimensional semantic vector having a shape of (N, T, C), where N is the batch size, T is the number of time steps, and C is the number of features.

[0026] Multiple convolutional layers are used on the input sequence to extract the features of time series data. Each convolutional layer uses different convolution kernels to capture different feature patterns. convolutional layers, and the output features of each convolutional layer are For the convolutional layers, the result of the convolution operation is: ,in, Indicates is a convolution operation and ReLU(·) is the activation function.

[0027] The output features of each convolutional layer All of them will go through the pooling layer to reduce the size of the feature map and retain the key features. The pooling operation uses the maximum pooling. The convolutional layer output is subjected to the maximum pooling operation: , where Pooling(·) represents the pooling operation.

[0028] By assigning different weights to the output features of each convolutional layer , and weighted sum them to fuse them into the final log template sequence comprehensive feature representation. The weighted fusion feature representation is: .in, Indicates The weight of the layer, is the feature after pooling. Represents the comprehensive features of the log sequence after multi-layer convolution and pooling operations. The final fusion feature It will be passed as input to the subsequent BiLSTM layer for further processing.

[0029] Furthermore, the feature sequence after convolution and pooling is input into the BiLSTM layer, and the global time series features in the log sequence are captured by the bidirectional LSTM, including the fusion features extracted from the CNN part. Indicates that the sequence is input to the BiLSTM layer, and the global time series features in the log sequence are captured through the bidirectional LSTM. In the forward LSTM, the input sequence flows from the first time step to the last time step, and the hidden state is gradually updated. The forward calculation formula is expressed as: ,in, Indicates that the forward LSTM is The hidden state of time steps, Indicates The input feature vector of time steps. In the backward LSTM, the input sequence flows from the last time step to the first time step, gradually updating the hidden state. The backward calculation formula is expressed as: ,in, represents the hidden state of the backward LSTM at the tth time step.

[0030] Furthermore, based on the features captured by the BiLSTM layer, the Attention mechanism is introduced to calculate the contribution of features at different moments in the log sequence to the anomaly detection results, including using a multi-head attention mechanism to calculate the correlation between each time step in the sequence to capture complex time series patterns. The attention calculation formula is expressed as: , where Q, K, and V are query, key, and value matrices respectively. is the dimension of the key vector. For multi-head attention, each head calculates an independent attention weight and then concatenates the outputs of all heads.

[0031] Then, bidirectional feature fusion is performed through a trainable parameter , dynamically adjust the fusion weights of forward and backward LSTM, the fusion calculation formula is: ,in It is The final hidden state of time steps is It is the fusion weight of the forward LSTM and the backward LSTM, and its value range is 0 to 1.

[0032] Then, the attention calculation is performed on the feature sequence after the bidirectional feature fusion to obtain the importance weight of each time step in the sequence. Weighted summation formula: ,in, Indicates The attention weights for time steps, It is the attention-weighted sequence representation.

[0033] Finally, the feature representation after attention calculation is regressed and predicted to generate the prediction result of anomaly detection. The prediction value of log anomaly detection is calculated using the fully connected layer , the prediction value formula .in, is the weight matrix of the fully connected layer, is the bias term. The mean square error is used as the loss function to calculate the error between the predicted value and the true value. The formula is: .in, is the true value, is the predicted value, is the sample size.

[0034] Furthermore, the trained model is used to infer the log data. If the difference between the predicted value and the actual value exceeds a set threshold, an alarm is triggered, including: Calculate the residual (i.e. the difference between the predicted value and the actual value) at each time point to form a residual sequence . Perform preprocessing such as denoising and smoothing on the residual sequence.

[0035] Choose a high initial screening threshold ,Residuals exceeding this preliminary screening threshold are considered as extreme values.,This preliminary screening threshold is determined through the experience of the ,operation and maintenance personnel or statistical analysis.

[0036] For those who exceed the initial screening threshold The residual is , by fitting the generalized Pareto distribution, the probability distribution of extreme events in the residuals is determined. The cumulative distribution function of the generalized Pareto distribution is expressed as: .in, is the generalized Pareto distribution function, and are the parameters fitted by maximum likelihood estimation (MLE).

[0037] After fitting the generalized Pareto distribution, a specific threshold is determined, and residuals exceeding this threshold are considered abnormal. This threshold is based on the specified confidence level For example, if the confidence level is set to 95%, 5% of the data are considered abnormal. Based on the generalized Pareto distribution model and the specified confidence level , calculate the threshold for anomaly detection : .here, is the initial screening threshold selected earlier, and are the parameters of the generalized Pareto distribution, is the confidence level.

[0038] For each residual exceeding the initial screening threshold , determine whether it exceeds the calculated anomaly detection threshold .if , then the data point is considered abnormal. , then the data point is considered normal.

[0039] When the system detects an anomaly, the alarm information is sent to the specified recipient list through communication channels such as email, SMS, and DingTalk messages.

[0040] On the other hand, the present invention also provides an automated log anomaly detection system based on deep learning, including: a log parsing module based on a large language model, a semantic feature extraction template, a model training module, and a log anomaly detection module; wherein: A log parsing module based on a large language model, used to extract representative log template sequences from raw log data; The semantic feature extraction module is used to encode the log template sequence using the Sentence-BERT model to generate a high-dimensional semantic vector expressing the log template; A model training module is used to construct a CNN-BiLSTM-Attention model, and adopt a supervised learning method to train and optimize the CNN-BiLSTM-Attention model using historically annotated normal and abnormal log data; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different times in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The log anomaly detection module is used to infer the log data to be tested through the trained CNN-BiLSTM-Attention model. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

[0041] In a third aspect, the present invention also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned automatic log anomaly detection method based on deep learning.

[0042] In a fourth aspect, the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned automatic log anomaly detection method based on deep learning.

[0043] The beneficial effects of the present invention are: The present invention utilizes a log parsing method based on a large language model to perform log parsing operations and extract a log template set, thereby reducing the redundancy and complexity of log data. Then, the log template is subjected to semantic feature extraction through the SBERT model to generate a high-dimensional semantic vector. Then, combining the advantages of the CNN-BiLSTM model, the log sequence data is subjected to preliminary feature extraction and dimensionality reduction through the CNN layer, and then the long dependency relationship of the time series is modeled by the BiLSTM layer. In order to further improve the model's ability to capture important feature information, the Attention mechanism is introduced into the BiLSTM model to accurately obtain the degree of influence of data at different times on the anomaly detection results. Finally, these features are input into the output layer of the model to generate log anomaly detection results. The method gives full play to the advantages of log template extraction capability based on a large model, semantic extraction capability of Sentence-BERT and temporal feature capture of the CNN-BiLSTM-Attention model, and significantly improves the accuracy and efficiency of log anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of an automated log anomaly detection method based on deep learning in the present invention; Figure 2 This is a flowchart for implementing the log parsing module based on a large language model; Figure 3 This is the implementation flow chart of the model training module; Figure 4 This is a block diagram of an automated log anomaly detection system based on deep learning in the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 As shown, the present invention provides an automated log anomaly detection method based on deep learning, comprising the following steps: Input the original log data into the log parsing module based on the large language model to extract representative log template sequences; Use the Sentence-BERT model to encode the log template sequence and generate a high-dimensional semantic vector expressing the log template; A CNN-BiLSTM-Attention model is constructed, and a supervised learning method is adopted to train and optimize the CNN-BiLSTM-Attention model using normal and abnormal log data with historical annotations; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different moments in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The trained CNN-BiLSTM-Attention model is used to infer the log data to be tested. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

[0047] like Figure 2 The log template extraction module based on the large language model of the present invention is specifically implemented as follows: The log template extraction module based on the large language model is mainly responsible for extracting log templates, which can convert the original unstructured log data into a structured format, thereby reducing the redundancy and complexity of the log data, and facilitating further anomaly detection of the logs.

[0048] This module first pre-processes the log messages of a set of raw log data through regular expressions to remove some variable contents in the logs, such as IP addresses, so as to prepare for subsequent log parsing operations. The regular expression that matches the IPv4 address in the log is: \b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b.

[0049] After preprocessing, a set of log data is obtained. Each piece of data in this set of log data is segmented by words or characters, the length of each log is calculated, and these different length values ​​are formed into a unique length set. The length of the log refers to the number of characters or words in this text.

[0050] Create a corresponding number of buckets based on the number of unique length sets. Each bucket is set up to store log messages of different lengths. Then distribute the logs to different buckets based on their lengths.

[0051] For log messages in the bucket, calculate the information entropy for each log

[0052] , Refers to a symbol in the log, including words or characters. is the probability of this symbol appearing in the log.

[0053] After calculating the information entropy value of each log, the logs are sorted from high to low according to these information entropy values. The logs with high entropy values ​​are placed at the front, and the logs with low entropy values ​​are placed at the back.

[0054] For each layer of logs, each log is checked in turn. The first word or character of each log is recorded as T(x).

[0055] For the first symbol T(x) of the current log, it has not been selected before (ie: T(x) ST(x)), then select this log and add it to the sample set S. Here ST represents the set of words or symbols that have been selected.

[0056] If some words or symbols in the current log have been selected for a predefined number of times, this log is selected and added to the sample set S.

[0057] This process will continue until the size of the selected sample set S reaches the predetermined number k, and the log selection process will stop.

[0058] For the selected log sample set, the Jaccard similarity method is used to calculate the similarity between these log samples. If the similarity is higher than a certain threshold, one of the log samples is deleted.

[0059] Select several log samples from the log message layer of each Bucket as log clustering centers , for each log message And each cluster center Calculating distance : , Indicates log messages Middle The symbol of the position; Represents the cluster center Middle The symbol of the position; Indicates that when , the value is 1, otherwise it is 0, according to each log message With each cluster center Distance , each log message Assign to the closest cluster center.

[0060] The log message of each cluster center is input into the large language model, and a context learning method is used to parse the log message and generate a log template. Context learning means that some sample log templates are used as context learning objects of the large language model. A log template means that if there is a log "User Zoey successfully logged in", the extracted log template is "User < > successfully logged in", where "Zoey" is a variable replaced by a wildcard < >. The cluster centers of the log template of the example and the log template to be generated The Prompt is input into the large language model, which automatically learns which parts of the log message to be parsed are fixed (i.e., part of the log template) and which parts are variables (i.e., log template < > part) to generate a log template and store it in the log template library.

[0061] For newly input log messages that do not match a log template in the log template library, the KNN algorithm is used to calculate the similarity between the log message to be parsed and the log messages in the log sample set and the generated log templates, and the most similar k log templates and log messages are selected to form a prompt input into the large language model, and the log template output by the large model is updated to the log template library. The prompt format example is as follows: Please extract the log template from the log message to be parsed according to the log message M, log template T, and log template generation principle P in these examples; example log templates T1, T2..., example log messages: M1, M2..., log template extraction principles: P1, P2..., log messages to be parsed: L1, L2...

[0062] Furthermore, the extracting semantic vectors from the log templates using the Sentence-BERT model includes: The Sentence-BERT model receives a series of log templates as input. When processing the log templates, each log template can be regarded as a sentence.

[0063] These input sentences are encoded by the BERT model. The BERT model is a deep bidirectional Transformer that can understand the contextual information in the sentence. During the encoding process, the BERT model converts each symbol in the sentence into a vector.

[0064] After BERT encoding, the Sentence-BERT model aggregates the vectors of each word to generate a vector representing the entire sentence.

[0065] like Figure 3 , the model training module of the present invention is specifically implemented as follows: The log semantic vector output by the semantic feature extraction module is input into the CNN layer to extract local features of the log sequence. The input data shape is (N, T, C), where N is the batch size, T is the number of time steps, and C is the number of features.

[0066] Multiple convolutional layers are used on the input sequence to extract the features of time series data. Each convolutional layer uses different convolution kernels to capture different feature patterns. convolutional layers, and the output features of each convolutional layer are For the convolutional layers, the result of the convolution operation is: ,in, Indicates is a convolution operation and ReLU(·) is the activation function.

[0067] The output features of each convolutional layer All of them will go through the pooling layer to reduce the size of the feature map and retain the key features. The pooling operation uses the maximum pooling. The convolutional layer output is subjected to the maximum pooling operation: , where Pooling(·) represents the pooling operation.

[0068] By assigning different weights to the output features of each convolutional layer , and weighted sum them to fuse them into the final feature representation. The feature representation after weighted fusion is: .in, Indicates The weight of the layer, is the feature after pooling. Represents the comprehensive features of the log sequence after multi-layer convolution and pooling operations. The final fusion feature It will be passed as input to the subsequent BiLSTM layer for further processing.

[0069] The fused features extracted from the CNN part Indicates that the sequence is input to the BiLSTM layer, and the global time series features in the log sequence are captured through the bidirectional LSTM. In the forward LSTM, the input sequence flows from the first time step to the last time step, and the hidden state is gradually updated. The forward calculation formula is expressed as: ,in, Indicates that the forward LSTM is The hidden state of time steps, Indicates The input feature vector of time steps. In the backward LSTM, the input sequence flows from the last time step to the first time step, gradually updating the hidden state. The backward calculation formula is expressed as: ,in, represents the hidden state of the backward LSTM at the tth time step.

[0070] Then, a multi-head attention mechanism is used to calculate the correlation between each time step in the sequence to capture complex time series patterns. The attention calculation formula is expressed as: , where Q, K, and V are query, key, and value matrices respectively. is the dimension of the key vector. For multi-head attention, each head calculates an independent attention weight and then concatenates the outputs of all heads.

[0071] Then, bidirectional feature fusion is performed through a trainable parameter , dynamically adjust the fusion weights of forward and backward LSTM, the fusion calculation formula is: ,in It is The final hidden state of time steps is It is the fusion weight of the forward LSTM and the backward LSTM, and its value range is 0 to 1.

[0072] Then, the attention calculation is performed on the feature sequence after the bidirectional feature fusion to obtain the importance weight of each time step in the sequence. Weighted summation formula: ,in, Indicates The attention weights for time steps, It is the attention-weighted sequence representation.

[0073] Finally, the feature representation after attention calculation is regressed and predicted to generate the prediction result of anomaly detection. The prediction value of log anomaly detection is calculated using the fully connected layer , the prediction value formula .in, is the weight matrix of the fully connected layer, is the bias term. The mean square error is used as the loss function to calculate the error between the predicted value and the true value. The formula is: .in, is the true value, is the predicted value, is the sample size.

[0074] Furthermore, the trained model is used to infer the log data. If the difference between the predicted value and the actual value exceeds a set threshold, an alarm is triggered, including: Calculate the residual (i.e. the difference between the predicted value and the actual value) at each time point to form a residual sequence . Perform preprocessing such as denoising and smoothing on the residual sequence.

[0075] Choose a high initial screening threshold ,Residuals exceeding this preliminary screening threshold are considered as extreme values.,This threshold is determined by the experience of the operation and ,maintenance personnel or statistical analysis.

[0076] For those who exceed the initial screening threshold The residual , by fitting the generalized Pareto distribution, the probability distribution of extreme events in the residuals is determined. The cumulative distribution function of the generalized Pareto distribution is expressed as: .in, is the generalized Pareto distribution function, and are the parameters fitted by maximum likelihood estimation (MLE).

[0077] After fitting the generalized Pareto distribution, a specific threshold is determined, and residuals exceeding this threshold are considered abnormal. This threshold is based on the specified confidence level For example, if the confidence level is set to 95%, 5% of the data are considered abnormal. Based on the generalized Pareto distribution model and the specified confidence level , calculate the threshold for anomaly detection : .here, is the initial screening threshold selected earlier, and are the parameters of the generalized Pareto distribution, is the confidence level.

[0078] For each residual exceeding the initial screening threshold , determine whether it exceeds the calculated anomaly detection threshold .if , then the data point is considered abnormal. , then the data point is considered normal.

[0079] When the system detects an anomaly, the alarm information is sent to the specified recipient list through communication channels such as email, SMS, and DingTalk messages.

[0080] On the other hand, Figure 4As shown, the present invention also provides an automated log anomaly detection system based on deep learning, and the modules included in the system can execute the above steps, specifically including: A log parsing module based on a large language model, used to extract representative log template sequences from raw log data; The semantic feature extraction module is used to encode the log template sequence using the Sentence-BERT model to generate a high-dimensional semantic vector expressing the log template; A model training module is used to construct a CNN-BiLSTM-Attention model, and adopt a supervised learning method to train and optimize the CNN-BiLSTM-Attention model using historically annotated normal and abnormal log data; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different times in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The log anomaly detection module is used to infer the log data to be tested through the trained CNN-BiLSTM-Attention model. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

[0081] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0082] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the various steps of the method of the present invention are implemented.

[0083] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automated log anomaly detection method based on deep learning, characterized in that: The following steps are involved: Input the original log data into the log parsing module based on the large language model to extract representative log template sequences; Use the Sentence-BERT model to encode the log template sequence and generate a high-dimensional semantic vector expressing the log template; A CNN-BiLSTM-Attention model is constructed, and a supervised learning method is adopted to train and optimize the CNN-BiLSTM-Attention model using normal and abnormal log data with historical annotations; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different moments in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The trained CNN-BiLSTM-Attention model is used to infer the log data to be tested. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

2. According to the deep learning-based automated log anomaly detection method of claim 1, it is characterized in that: The raw log data is input into the log parsing module based on the large language model to extract a representative log template sequence, including: Preprocess the input raw log data through regular expressions to remove variable content; Calculate the length of each log for the preprocessed raw log data, and form a unique length set from the length value of each log; Create a corresponding number of buckets based on the number of unique length sets. Each bucket is used to store log messages of different lengths. Log messages are allocated to different buckets based on their lengths. Calculate the information entropy for each log message in the bucket : , where Represents a symbol in a log message, including a word or character. represents the probability of this symbol appearing in the log, and i represents the symbol index in the log message; Sort log messages according to their information entropy values ​​from high to low; Divide the sorted log messages into multiple layers, check each log message in each layer, and record the first symbol of each log message as T(x); if the first symbol T(x) of the current log message has not been selected before, or the symbol included in the current log message has been selected a predefined number of times, add the current log to the sample set S; when the size of the sample set S reaches the predetermined number k, the log message selection process terminates; Select several log samples from the log message layer of each Bucket as log clustering centers , for each log message And each cluster center Calculating distance : , Indicates log messages Middle The symbol of the position; Represents the cluster center Middle The symbol of the position; Indicates that when , the value is 1, otherwise it is 0, according to each log message With each cluster center Distance , each log message Assign to the nearest cluster center; The log messages of each cluster center are input into the large language model, and the log messages are parsed based on the context learning method to generate a set of log templates.

3. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The Sentence-BERT model is used to encode the log template sequence to generate a high-dimensional semantic vector expressing the log template, including: The Sentence-BERT model receives a sequence of log templates and treats each log template as a sentence; Encode each symbol in the sentence into a vector; The vectors of each sentence are aggregated to generate a high-dimensional semantic vector representing the entire sentence.

4. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The CNN layer receives the high-dimensional semantic vector expressing the log template, extracts the local features of the log template sequence, and optimizes the feature expression through convolution and pooling operations to obtain the comprehensive features of the log template sequence, including: The input high-dimensional semantic vector is recorded as (N, T, C), where N is the batch size, T is the number of time steps, and C is the number of features. Multiple convolutional layers are used to extract temporal features. Each convolutional layer uses different convolution kernels to capture different feature patterns. Pooling layer operations are performed on the temporal features output by each convolutional layer, and different weights are assigned. All output temporal features are weighted summed and fused into the comprehensive features of the log template sequence.

5. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The method of inputting the comprehensive features of the log template sequence into the BiLSTM layer and capturing the global time series features in the log template sequence through the bidirectional LSTM includes: inputting the comprehensive features of the log template sequence into the BiLSTM layer, and in the forward LSTM, the comprehensive feature sequence of the log template sequence flows from the first time step to the last time step, and gradually updates the hidden state; in the backward LSTM, the comprehensive feature sequence of the log template sequence flows from the last time step to the first time step, and gradually updates the hidden state.

6. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The contribution of the global time series features at different moments in the log template sequence to the anomaly detection result based on the Attention layer is calculated as follows: Use a multi-head attention mechanism to calculate the correlation between each time step in the sequence; Through a trainable parameter ,Dynamically adjust the fusion weights of forward and backward LSTM to perform bidirectional feature fusion; Attention calculation is performed on the feature sequence after bidirectional feature fusion to obtain the contribution weight of each time step in the log template sequence.

7. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The difference between the predicted value and the actual value of each data point outputted through the output layer includes performing regression prediction on the feature representation after attention calculation, generating a predicted value for anomaly detection, and using the mean square error as a loss function to calculate the error between the predicted value and the actual value.

8. The method for automatic log anomaly detection based on deep learning according to claim 1, characterized in that: The trained CNN-BiLSTM-Attention model is used to infer the log data to be tested. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered, including: Calculate the difference between the predicted value and the actual value at each time point to form a residual sequence , denoise and smooth the residual sequence; Select a preliminary screening threshold, extract the residuals exceeding the preliminary screening threshold, and perform a generalized Pareto distribution fitting on the residuals exceeding the preliminary screening threshold to obtain the parameters of the generalized Pareto distribution, including the shape parameter and scale parameter ; Based on the generalized Pareto distribution fitting results and the specified confidence level , calculate the threshold for anomaly detection ; For each residual that exceeds the initial screening threshold, determine whether it exceeds the threshold for anomaly detection ,if , then the data point is considered abnormal; if it does not exceed, then the data point is considered normal; When the system detects an anomaly, it sends an alert to a designated list of recipients via communication channels including email, SMS, and DingTalk messages.

9. An automated log anomaly detection system based on deep learning, characterized in that: include: A log parsing module based on a large language model, used to extract representative log template sequences from raw log data; The semantic feature extraction module is used to encode the log template sequence using the Sentence-BERT model to generate a high-dimensional semantic vector expressing the log template; A model training module is used to construct a CNN-BiLSTM-Attention model, and adopt a supervised learning method to train and optimize the CNN-BiLSTM-Attention model using historically annotated normal and abnormal log data; wherein the CNN-BiLSTM-Attention model includes a CNN layer, a BiLSTM layer, an Attention layer, and an output layer, wherein the CNN layer receives a high-dimensional semantic vector expressing a log template, extracts local features of a log template sequence, and optimizes feature expression through convolution and pooling operations to obtain comprehensive features of the log template sequence; the comprehensive features of the log template sequence are input into the BiLSTM layer, and the global time series features in the log template sequence are captured through a bidirectional LSTM; based on the Attention layer, the contribution of the global time series features at different times in the log template sequence to the anomaly detection result is calculated; and the difference between the predicted value and the actual value of each data point is output through the output layer; The log anomaly detection module is used to infer the log data to be tested through the trained CNN-BiLSTM-Attention model. If the difference between the predicted value and the actual value exceeds the set threshold, an alarm is triggered.

10. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Among them, when one or more programs are executed by the one or more processors, the one or more processors implement the deep learning-based automated log anomaly detection method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement an automated log anomaly detection method based on deep learning as described in any one of claims 1-8.

Images