Log anomaly detection method, system and electronic device

Abstract

The application discloses a log anomaly detection method and system and electronic equipment, and relates to the technical field of log detection.The application can accurately identify and extract key information in logs by performing structural processing and pattern recognition on preprocessed log data.In the log analysis process, the application adopts a log analysis algorithm based on a fixed depth tree, can ignore variable data in logs, maximally clusters logs with similar prefix tokens, and can also classify a small part of logs that do not meet preset requirements into correct log groups, thereby improving the accuracy and robustness of log analysis.Furthermore, by constructing triplets of log events, randomly replacing original log data with triplets, and performing negative sampling, the robustness and generalization ability of an anomaly detection model are enhanced, the log data distribution in a real scene can be better simulated, and the performance and reliability of log anomaly detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log detection, in particular to a log anomaly detection method and system and electronic equipment. BACKGROUND

[0002] With the rapid development of information technology, large-scale distributed systems are widely used in cloud computing, network security, finance and other fields. These systems can often handle large amounts of data, support high concurrency access, and implement complex functions and business processes, bringing great convenience to people's daily life. However, as the size of the system becomes larger and the architecture becomes more complex, the possibility of system vulnerabilities increases, and when the system is abnormal, it becomes more difficult to locate and solve the system exception.

[0003] Logs are important information for recording the running state of the system and are an important basis for system operation and maintenance personnel to locate and analyze system exceptions. Traditional log anomaly detection methods usually rely on rules and feature engineering, which requires manual definition of various rules and patterns to identify abnormal events. However, these methods have some challenges. First, manually defining rules and patterns requires the knowledge of domain experts, which is a very tedious and error-prone process. Second, due to the complexity and diversity of log data, it is difficult for technical personnel to manually design rules and patterns to completely capture all abnormal situations. Moreover, due to the dynamic nature of the system and the continuous growth of log data, technical personnel must update and adjust the rule base in a timely manner, which all leads to the fact that traditional methods are not suitable for today's needs.

[0004] In recent years, the rapid development of artificial intelligence technology has provided new solutions for log anomaly detection. Some researchers have applied machine learning techniques to log anomaly detection, such as log anomaly detection methods based on invariant mining, principal component analysis and decision trees. These methods have certain advantages in accuracy and efficiency compared to traditional methods, but they cannot extract high-level features from logs and are difficult to analyze deep hidden association information between logs.

[0005] With the development of deep learning technology, more and more deep learning-based log anomaly detection methods are proposed, such as DeepLog, LogAnomaly, Logsy, etc. The deep learning-based method usually includes four steps: first, the system log is collected using a log collection tool, then the collected log is parsed, and the semi-structured log message is parsed into a structured log template. Log parsing is an important step in log anomaly detection, and its accuracy determines the accuracy of log anomaly detection. Second, features are extracted from the parsed log, usually converting the log template into a template vector. Finally, a deep learning model is constructed to detect anomalies in the template. Although the current deep learning method has achieved good results, it often performs unstable and robust in practice. Existing methods usually use open-source log parsing algorithms such as Drain and Spell, which still have room for improvement in speed and accuracy. At the same time, existing methods usually use a single deep learning network model such as RNN and LSTM, which is difficult to fully utilize all the information of the log. In addition, log data is a typical unbalanced data, and labeling log data is a very time-consuming and laborious engineering. SUMMARY

[0006] To solve the above problems existing in the prior art, the present application provides a log anomaly detection method, system and electronic equipment.

[0007] To achieve the above purpose, the present application provides the following scheme:

[0008] A log anomaly detection method comprises:

[0009] Obtain raw log data and preprocess the raw log data to obtain log data; the raw log data includes timestamp, log level and log content;

[0010] The log data is parsed using a log parsing algorithm based on a fixed depth tree to obtain a log event template;

[0011] A triple dictionary is constructed according to the log event template, and negative sampling is performed on the log in the log data based on the triple dictionary to obtain a log sequence;

[0012] Based on a pre-trained language model, the log sequence is converted into a log sequence embedding vector;

[0013] An anomaly detection classifier based on a convolutional neural network and a gated recurrent unit is constructed to perform anomaly detection on the log sequence embedding vector.

[0014] Optionally, the preprocessing includes word segmentation processing and case conversion processing.

[0015] Optionally, the log parsing algorithm of the fixed-depth tree comprises setting a parameter variable, text preprocessing, searching according to a log length, searching according to a constant token, and similarity calculation.

[0016] Optionally, the log in the log data is negatively sampled based on the triple dictionary, and specifically comprises:

[0017] All logs in the log data are traversed, and the triple of the log event appearing in the logs is counted to obtain a triple frequency of appearance;

[0018] The triple of the log event in the log is replaced based on the triple frequency of appearance, and the negative sampling is completed.

[0019] Optionally, the pre-trained language model is a word2vec model.

[0020] According to the specific embodiments of the present application, the following technical effects are provided:

[0021] The log anomaly detection method provided by the present application can accurately identify and extract key information in the log by performing structured processing and pattern recognition on the preprocessed log data. In the log parsing process, the log parsing algorithm based on the fixed-depth tree is used, which can ignore the variable data in the log, maximize the clustering of logs with similar prefix tokens, and at the same time, a small part of logs that do not meet the preset requirements can also be classified into the correct log group, thereby improving the accuracy and robustness of log parsing. Moreover, by constructing the triple of the log event, random triple replacement is performed on the original log data to complete negative sampling, thereby enhancing the robustness and generalization ability of the anomaly detection model, better simulating the log data distribution in the real scene, and improving the performance and reliability of log anomaly detection.

[0022] Further, the present application also provides the following implementation structure:

[0023] A log anomaly detection system applied to the log anomaly detection method provided above, wherein the system comprises:

[0024] A preprocessing module is configured to obtain original log data and preprocess the original log data to obtain log data, wherein the original log data comprises a timestamp, a log level, and log content.

[0025] A log template parsing module is configured to parse the log data by using a log parsing algorithm based on a fixed-depth tree to obtain a log event template.

[0026] A log sequence generation module is configured to construct a triple dictionary according to the log event template, and negatively sample a log in the log data based on the triple dictionary to obtain a log sequence.

[0027] a vector conversion module configured to convert the log sequence into a log sequence embedding vector based on a pre-trained language model;

[0028] an anomaly detection module configured to construct an anomaly detection classifier based on a convolutional neural network and a gated recurrent unit, and perform anomaly detection on the log sequence embedding vector.

[0029] An electronic device comprises:

[0030] a memory configured to store a computer program;

[0031] a processor connected to the memory and configured to invoke and execute the computer program to implement the log anomaly detection method provided above.

[0032] Optionally, the memory is a computer readable storage medium.

[0033] The technical effects achieved by the two implementation structures provided by the present application are the same as those achieved by the log anomaly detection method provided above, and therefore will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0035] Figure 1 a log anomaly detection method flowchart provided by the present application;

[0036] Figure 2 a log anomaly detection method implementation architecture diagram provided by the present application;

[0037] Figure 3 a CBOW model structure diagram provided by the present application;

[0038] Figure 4 a Skip-gram model structure diagram provided by the present application;

[0039] Figure 5 a fixed depth tree structure diagram provided by the present application;

[0040] Figure 6 a negative sampling flowchart provided by the present application;

[0041] Figure 7 a log grouping diagram provided by the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0043] The present application aims to provide a log anomaly detection method, system and electronic device, which can detect log anomalies more accurately and quickly, improve the performance and reliability of log anomaly detection, and solve the problem of data imbalance.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0045] As shown in Figure 1 and Figure 2 The log anomaly detection method provided by the present application comprises:

[0046] Step 100: Obtain original log data and pre-process the original log data to obtain log data. The original log data includes timestamp, log level and log content.

[0047] In actual application process, the original log data is collected by the collector as the basis for anomaly detection.

[0048] Further, considering that the formats of logs of different platforms are not unified, some are all in lowercase and some are mixed case, in the process of text preprocessing of the original log data, word segmentation, case conversion, etc. can be included. For example, for a log "Verification succeeded for blk_123", by converting all the capital letters to lowercase, "verification succeeded for blk_123" can be obtained. Word segmentation refers to dividing a log into several words, and for the above log, for example, by word segmentation, verification, succeeded, for, blk_123 can be obtained, so as to facilitate subsequent log parsing and feature representation.

[0049] In this step, the original log cannot be directly detected for anomaly, and invalid data, duplicate data and missing data need to be removed, and the log needs to be segmented and converted to lowercase. This step aims to prepare for subsequent log parsing and anomaly detection.

[0050] Step 101: parse the log data using a fixed-depth tree-based log parsing algorithm to obtain log event templates. Logs are usually unstructured text data, and this process involves structuring the text data to accurately identify and extract key information from the logs, constructing structured log events. These log events can be used as input for subsequent anomaly detection models.

[0051] The fixed-depth tree-based log parsing algorithm used in the present application includes five steps: setting parameter variables, text preprocessing, searching by log length, searching by constant token, and similarity calculation. The purpose is to extract log event templates. This algorithm structures the log data into log events with certain semantics by identifying and matching fixed log templates, keywords, and variables. Specifically:

[0052] 1) Set parameter variables: set some necessary parameters in the algorithm, such as setting the (maximum) tree depth and similarity threshold.

[0053] 2) Text preprocessing: text preprocessing aims to pre-replace common parameters in the log, which facilitates subsequent log parsing. For example, type replacement is usually required before log parsing, i.e. predefining some regular expressions to replace special symbols in the original log data, such as replacing IP addresses in the log with the word IP, etc. After replacement, the log needs to be reduced, reducing multiple consecutive spaces to one space.

[0054] 3) Search by log length: current log parsing algorithms, such as Drain, are usually based on the assumption that log messages with the same log event may have the same log message length, so this algorithm also searches by log length. Log length is the sum of all tokens (constants and variables) in a log, which can be used to distinguish log messages of different lengths and facilitate the aggregation of similar events. For example, logs with different log word lengths are divided into different nodes. Specifically, according to the length of the current log, find out whether there is a corresponding length node in the first layer node. If found, search the corresponding length node, otherwise create a new length node.

[0055] 4) Constant token search: Different from the length layer node of the first layer, when searching from the second layer, only constant tokens are considered for searching, and the constant tokens of the log message are sequentially matched with the nodes. If the corresponding node is matched, the next layer node is searched, otherwise a new node is created. Specifically, according to the word order of the log, the constant token is searched, that is, only the constant in the log is considered without considering the variable. This step is an improvement of the existing mainstream algorithm Drain, because the number of variable words in the log is large, it is not convenient to correctly parse the log, therefore only constant words are considered in this step.

[0056] 5) Similarity calculation: The log is traversed from the first word to the last word or reaches the preset maximum tree depth, and the traversal ends. The log reaches the last log event template layer, and the similarity between the log and the existing template is calculated to determine the final log event template. If the similarity is greater than the threshold, it is considered that the log also has this log template, if it is less than or equal to the threshold, the Cartesian distance of the minhash of the log and the log group within a certain range of length is compared, and it is judged whether the Cartesian distance reaches the threshold. If it reaches the threshold, the corresponding log group is added. This step is also the innovation point of the algorithm. Drain algorithm creates a new template after the first similarity determination fails. This algorithm considers that even if two logs are different in length, they may have the same template if the lengths are similar. Therefore, the logs within a certain range of length are taken out for comparison. If both similarity determinations fail, a new log group is created, and the existing logs need to be completely replaced with placeholders. Finally, the newly created log category is added to the prefix search tree. The newly created template object contains the template content (which is the content of the log itself), the corresponding log id and other parameters, and the template object is added to the log group, so that the following log can also try to match the new template. The newly created template does not affect the old template.

[0057] Specifically, when the log cluster bucket at the bottom of the tree is traversed, the similarity between the current log and the corresponding log is calculated, and the calculation formula of the similarity is that whether two tokens are the same is compared from left to right, if the same, the counting variable is added by one, and finally divided by the total length to obtain the similarity. If the similarity reaches the threshold, the current log is added to the log group, if not similar, the log is compared with all the log groups within a certain range in front and back, the Cartesian distance of the minhash of these logs and the current log group is compared, and whether the similarity reaches the threshold is judged. If the threshold is reached, it means that the log matches the corresponding log template, and there is a parameter of the log id corresponding to the log template in the log template object, and the log id is appended to the back of the parameter, so that the logs corresponding to the template can be obtained by querying the parameter of the template, without adding the log itself. If the log is compared with the log template of the same length, the log id is added, and the current template is also updated. The update method is as follows:

[0058] The words of the log and the event template are traversed from left to right, if the same, the word is kept. Otherwise, the word is updated in the log event through the wildcard *. For example, the original template is ab c d e. The new log ab c d f matches successfully (reaches the threshold), and the template is updated to ab cd*, and the wildcard * represents any word.

[0059] Note that the update here only occurs when the length of the log is the same, and when the log group within a certain length range is taken out for minhash comparison, even if the similarity threshold is reached, the original template will not be updated, and only the log id will be added to the template object parameter.

[0060] The matching process of comparing the Cartesian distance of the minhash of these logs and the current log group to judge whether the similarity reaches the threshold is as follows: if the length of the log is 7, the log group of the template of the length of 7 is traversed, and the similarity comparison is performed with all the templates of the log group. If the threshold is not reached, it means that no matching is found (the comparison method here is to compare each word from left to right). Then the log group of the template of the length of 6 is matched, and if the matching is found, it means that the template of the log is the matched template. If no matching is found, the log group of the template of the length of 8 is compared.

[0061] If both kinds of similarity judgment fail, a new log group is created, and at this time, complete placeholder replacement is required for the existing logs, and finally the created new log category is added to the prefix search tree.

[0062] The fixed tree (the structure of which is as follows Figure 5The reason is that the length of the log is uncertain, some are long, some are short, if each word of each log is traversed and searched, a very deep tree may be obtained. Therefore, based on the fixed depth tree (also the data structure adopted by the mainstream algorithm Drain), a maximum tree depth is set, so for the log, if the log has been traversed before the maximum depth, it directly enters the log template layer, if it has not been traversed when it reaches the preset depth, it does not perform subsequent traversal and directly enters the template layer.

[0063] Step 102: Constructing a triple dictionary according to the log event template, and based on the triple dictionary, negative sampling is performed on the logs in the log data to obtain a log sequence. In view of the positive and negative sample imbalance of the log data itself, a negative sampling strategy is adopted to balance the distribution of normal and abnormal log events. By increasing the diversity and complexity of the data, the robustness and accuracy of the model can be improved, and the abnormal detection performance can be further improved.

[0064] In actual application process, the sampling process of negative sample is as Figure 6 shown, which includes:

[0065] 1) Traverse the log training set to generate log event triples: first, traverse the log training set to obtain each log data, and convert it into a log event triple form. Each triple consists of a timestamp, an event template and a variable, which is used to represent the structure and content of the log data. First, the system traverses the log training set.

[0066] 2) Construct a log event triple dictionary: based on the log event triples obtained by traversal, a dictionary structure is constructed, in which the key is the triple and the value is the frequency of the triple in the training set. By constructing the triple dictionary, the occurrence frequency of each triple can be quickly queried and counted, which provides a basis for subsequent negative sampling. Among them, the triple is regarded as a triple of three continuous words, and a large number of logs are obtained in the data set. By traversing each log, the triple count of each log can be obtained, and the sum is summed to obtain the corresponding number of times of each triple, that is, the obtained triple dictionary.

[0067] Taking the log "verification succeeded for blk_123" as an example, the dictionary {(verification succeeded for):1,(succeeded for blk_123):1} is obtained after traversal, which contains two key-value pairs, respectively representing (verification succeeded for) and (succeeded for blk_123) appear once.

[0068] For example, the dataset has 4 logs: 1: ab c d e 2: ab c e f 3: ab 4: e fo dp o c. The first log has three triplets (a, b, c) and (b, c, d) and (c, d, e), the second also has three: (a, b, c), (b, c, e), (c, e, f), the third is less than 3 in length, so the triplets are (a, b, ). The third position of the triplet is empty, and the fourth has five: (e, f, o), (f, o, d), (o, d, p), (d, p, o), (p, o, c), so the summary sum can get a triplet dictionary, which is:

[0069] {

[0070] (a, b, c): 2 (meaning that (a, b, c) appears twice)

[0071] (b, c, d): 1

[0072] (c, d, e): 1

[0073] … (the rest of the triplets are 1 times)

[0074] }

[0075] Through this step, the frequency of triplets appearing in the normal log can be obtained.

[0076] 3) Select a normal log sequence and perform a random replacement operation on the normal sequence by log time triplets: To solve the problem of unbalanced positive and negative log samples, a normal log sequence is selected and a random replacement operation is performed on the sequence. Specifically, the system replaces some log events in the normal log sequence with other lower frequency log event triplets by using the frequency information in the log event triplet dictionary. In this way, while preserving the overall structure of the normal log sequence, a certain proportion of abnormal log events can be introduced to increase the number of abnormal samples.

[0077] For example, there is a normal log consisting of five words: a, b, c, d, e. By random replacement, such as replacing (f, a, c) in the triplet dictionary with (b, c, d), the log becomes a, f, a, c, e after replacement. At this time, the frequency of the triplet event in the log in the triplet dictionary is lower than a threshold value, so the log is a negative sample successfully obtained.

[0078] 4) Generate an abnormal sequence: After the random replacement operation of the normal log sequence, the system generates a log sequence that has been negatively sampled, which contains a certain proportion of abnormal log events. The abnormal sequence obtained in this way can be used to train and evaluate the anomaly detection model, improving the ability to recognize abnormal events.

[0079] Step 103: Based on the pre-trained language model, the log sequence is converted into a log sequence embedding vector. This step, based on the convolutional neural network and the gated recurrent unit, constructs an efficient and accurate anomaly detection model. This model fully utilizes the advantages of CNN in feature extraction and the capabilities of GRU in sequence modeling, enabling effective identification and classification of abnormal log events and improving the performance and accuracy of the anomaly detection model.

[0080] In practical applications, the log sequence is converted into a log sequence embedding vector based on the pre-trained language model word2vec. The log sequence here contains all sample sequences, which include a large number of positive samples and a small number of negative samples.

[0081] Word2vec is a tool for word vector calculation developed by Google. It is a tool for calculating word vectors and a language algorithm model based on CBow and skip-gram models to calculate word vectors. The structure is shown in the figure, CBOW structure is based on the context to predict the current word, while Skip-gram is to predict the surrounding words given the current word. Word2vec successfully trains over a billion texts, which can convert words in log sequences into numerical vectors for subsequent anomaly detection.

[0082] Specifically, the logs need to be grouped first, and a fixed time window is used. The log is a text data with a timestamp, and by setting a window size, such as one second or one minute, the logs in this time window are a group to facilitate logical analysis of the log group. The window-based log grouping technique used is shown in Figure 7 The figure shows sliding time window, fixed time window, and session window, respectively. The fixed window and sliding window are based on timestamps, and the fixed window has a fixed time span or duration Δt, and the window size is a constant value, such as one second or one minute. All logs in this time window are classified as log sequences. Unlike the fixed window, the sliding window consists of two attributes: window size and step size, such as a one-hour window that slides every five minutes. In addition, the session window is based on identifiers (a or b) rather than timestamps. In each sequence, all logs share a common identifier, indicating that they come from the same task operation. The common log grouping methods are the above three, among which the fixed time window logs can only appear in one group. A log may appear in multiple windows in the sliding time window, and the sliding event window has a finer granularity but more complex calculations. The session window is based on identifiers, and all logs in a log group have the same identifier.

[0083] After grouping the logs, vectorization is performed using word2vec, which is a pre-trained language model. It can capture the contextual information and semantic relationships in the log sequence and convert the log data into vector representations. Word vectors are calculated based on CBow and skip-gram models. The structures of CBow and skip-gram models are shown in Figure 3 and Figure 4 where CBOW structure predicts the current word based on the context, and Skip-gram predicts the surrounding words given the current word.

[0084] word2vec successfully trains over a billion levels of text, which can convert words in the log sequence into numerical vectors for subsequent anomaly detection. After obtaining the vector of each word in the log, the importance of each word in each log can be obtained through TF-IDF. For a log, the vector representation of the log corresponding to this log can be obtained by multiplying the TF-IDF value of each word with the corresponding word2vec vector and adding all the word vectors.

[0085] In the present application, word2vec only converts each word in the log into a vector, and each log (which can be regarded as a sentence) needs to be converted into a vector. Here, the TF-IDF algorithm is used. TF-IDF (Term Frequency-Inverse Document Frequency) is a commonly used weighting technique for information retrieval and data mining. TF is Term Frequency, and IDF is Inverse Document Frequency. TF-IDF is a statistical method to evaluate the importance of a word to a file set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a file, but decreases in inverse proportion to the frequency of its appearance in the corpus. Through TF-IDF, the importance of each word in each log can be obtained. For a log, the vector representation of the log corresponding to this log can be obtained by multiplying the TF-IDF value of each word with the corresponding word2vec vector and adding all the word vectors.

[0086] Step 104: Construct an anomaly detection classifier based on convolutional neural network and gated recurrent unit, and perform anomaly detection on the log sequence embedding vector.

[0087] The structure of the anomaly detection classifier based on convolutional neural network (CNN) and gated recurrent unit (GRU) constructed in the present application is shown in Figure 2The abnormality detection classifier includes a CNN, GRU feature extraction, an attention mechanism feature fusion, and two fully connected + dropout operations, and finally outputs an abnormality detection result through a softmax.

[0088] The model can implement the whole process of abnormality detection. Specifically, the obtained word vectors are respectively input into the CNN and the GRU. For the obtained data, a weight vector is obtained through feature fusion based on the attention mechanism, and then the fusion features are obtained through weighting. For the features, two fully connected and dropout operations are used. Finally, the result is input into a softmax function to generate a probability score, and the threshold is judged to obtain the result of whether the log is abnormal.

[0089] In the above abnormality detection classifier, the CNN can extract high-level abstract features from local features, and the GRU can model and capture long-term dependencies for sequence data. The combination of the two models can effectively perform abnormality detection on log vectors.

[0090] The application also uses an attention mechanism to fuse the two features. When processing sequence data (such as text, speech, time series, etc.), the attention mechanism is a commonly used technique. The attention mechanism allows the model to assign different weights to input information at different time steps or positions when processing sequence data, dynamically focusing on the most useful part for the current task. The introduction of the attention mechanism allows the model to adaptively focus on important input information, rather than simply treating all inputs as equal weights. Through attention mechanism fusion, different features can be adaptively weighted, improving the expression ability and robustness of the model. After fusing the features, two tanh fully connected layers and dropout layers are used. The fully connected layer helps to map the concatenated features to a higher dimensional space, so as to better distinguish different logs. Using dropout can reduce the risk of overfitting. Finally, the result processed by the fully connected layer is input into the Softmax function, which will generate a probability score. If it is greater than the threshold, it is considered as an abnormal log, otherwise it is considered as a normal log. For example, when the probability of the abnormal class is greater than a certain threshold (such as 0.5), it is determined as an abnormal log, otherwise it is determined as a normal log, to identify potential abnormal events.

[0091] In actual application, the CNN extraction in the application is a general term describing the process of inputting data into the CNN and obtaining output data. In fact, the extraction of CNN and GRU is performed by using open source code. Specifically, the specific process of data processing by CNN and GRU is as follows:

[0092] A, The specific process of CNN processing data: the extraction of CNN is obtained through convolution, activation function and pooling function.

[0093] 1) Convolution operation: In the present application, the convolution operation is applied to the vector representation of the log sequence, and the convolution operation is calculated by sliding the convolution kernel on the input data. In the present application, the input data is usually the vector representation of the log sequence. The convolution kernel is a small weight matrix, and its size is usually much smaller than the input data. Convolution operation refers to the sliding of the convolution kernel on the input data, element-wise multiplication and accumulation, to generate a new feature value. In the convolution process, the convolution kernel is multiplied element-wise with the local region of the input data, and then the product is added to generate a new feature value. By sliding the convolution kernel, features at different positions can be extracted on the input data. This process is similar to the receptive field in the visual cortex, that is, the visual system only focuses on a small part of the visual field to extract local features when observing objects. Specifically:

[0094] First, the vector representation of the log sequence is convolved by a series of convolution kernels of different sizes. Each convolution kernel corresponds to a specific window size, similar to different sizes of receptive fields. During convolution, the convolution kernel slides over the vector representation, element-wise multiplies the vectors within each window, and sums them to generate a new feature value. This operation can be understood as feature extraction on local segments of different lengths, capturing local patterns at different levels, such as different time scales of event patterns that may exist in the log sequence. By applying multiple convolution kernels, rich local feature information can be obtained.

[0095] 2) Activation function: After the convolution operation, an activation function is applied to the output feature map, such as the ReLU (Rectified Linear Unit) activation function, which maps each element in the feature map to a non-linear function (i.e., introduces non-linearity). The purpose of this is to introduce non-linear capabilities in the feature extraction process, allowing the network to learn more complex patterns and features. Specifically, after convolution and weighted summation, the ReLU function is applied to each feature value, setting negative values to zero. This step introduces a non-linear transformation, allowing the network to learn more complex local patterns and abstract features, thereby better distinguishing different log patterns, allowing the network to better model the complexity of the input data.

[0096] 3) Pooling operation: After extracting local features, a pooling operation is used to reduce the spatial size of the feature map and decrease the computational load while preserving the main information (i.e., key information). The pooling operation used in this invention is MaxPooling, which selects the maximum value in each pooling window as the output feature value. MaxPooling is a commonly used pooling operation that selects the most significant feature value from each region as the representative. By MaxPooling, the data dimension is reduced while maintaining important information, which helps to reduce the computational burden and improve the generalization ability of the model. Pooling operation helps to improve the translational invariance of the model, that is, no matter where the feature appears in the image or text, the model can recognize it. This is very useful for realizing high-level abstraction of local features.

[0097] Through multiple stacking of convolutional layers, activation functions and pooling layers, the CNN model gradually extracts higher-level and more abstract feature representations from the input local features. The bottom convolutional layer usually focuses on low-level patterns and captures some low-level local features such as single event fragments. As the number of layers increases, information is passed to deeper convolutional layers, and features become more abstract, such as object parts or semantic features in text. At this time, the model can gradually capture higher-level abstract features such as complex event patterns, behavior sequences, etc. Finally, these high-level abstract features are mapped to the final classification result through the fully connected layer.

[0098] B, the specific process of GRU processing data:

[0099] In this invention, GRU is used as a key feature extraction method. GRU consists of a reset gate (Reset Gate) and an update gate (Update Gate), which adaptively determine how to weight the input and when to update and reset the hidden state through the training process.

[0100] 1) Initialize the hidden state: First, initialize the hidden state of GRU, usually a zero vector. This hidden state will be updated constantly when processing the entire sequence, carrying the information of the sequence.

[0101] 2) Sequence traversal: For the input log matrix, it is input into the GRU model step by step. At each time step, GRU calculates the new hidden state of the current time step according to the input of the current time step and the hidden state of the previous time step. This hidden state contains the information of the input sequence before the current time step, thus capturing the timing information.

[0102] 3) Gating mechanism calculation: When calculating the hidden state, the reset gate controls how much information to forget from the previous hidden state at the current time step, while the update gate controls how much new information to incorporate from the current input at the current time step. This gating mechanism allows the GRU to adaptively balance between long-range dependencies and short-range dependencies, thus better modeling sequence relationships.

[0103] Update Gate: According to the input of the current time step and the hidden state of the previous time step, the input vector of the current time step is spliced with the hidden state of the previous time step, and an update gate is obtained through an activation function, which is used to control the updated information. Calculate the update gate. The update gate determines whether to retain the information of the hidden state of the previous time step at the current time step.

[0104] Reset Gate: By using a fully connected layer, the input vector of the current time step is spliced with the hidden state of the previous time step, and then an activation function is used to obtain a reset gate, which is used to control whether to forget the previous information. Calculate the reset gate to control whether to reset the hidden state of the previous time step at the current time step.

[0105] 4) Hidden state update: Using the calculated update gate and reset gate, combine the input of the current time step and the hidden state of the previous time step to calculate the new candidate hidden state.

[0106] 5) Current hidden state calculation: Through a weight related to the update gate, the new candidate hidden state is linearly combined with the hidden state of the previous time step to obtain the hidden state of the current time step. This hidden state contains the input information of the current time step and the memory of the previous time step.

[0107] 6) Output generation: After completing the sequence traversal, the GRU will obtain a series of hidden states, each of which represents the feature representation of the corresponding time step in the input sequence. According to application requirements, all hidden states can be selected for subsequent tasks, or only the last hidden state can be used as the feature representation of the entire sequence.

[0108] Through this series of calculations, the GRU can gradually capture the timing patterns and context information in the input log vector matrix. This processing process fully considers the timing properties of the sequence, allowing the model to better model and analyze log data for subsequent text anomaly detection tasks.

[0109] In the above process, the fusion is achieved through attention mechanism-based fusion. Specifically:

[0110] 1) Dimension alignment: First, the output vector of CNN and the output matrix of GRU need to be dimensionally aligned so that they can be weighted and fused in the attention mechanism. The invention uses a fully connected layer to map the output vector of CNN to the same dimension as the output matrix of GRU, so that the output vectors of CNN and GRU have the same dimension for subsequent weight fusion.

[0111] 2) Attention weight calculation: Calculate the attention weight between the outputs of CNN and GRU using the attention mechanism. The invention uses dot product attention method. This method measures the similarity between two vectors based on their dot product (inner product). Assuming there are two vectors A and vector B, the calculation formula of dot product attention (A, B) is: (A, B) = A·B. Where A·B represents the inner product of vector A and vector B.

[0112] The calculation process of dot product attention is equivalent to aligning the directions of two vectors in the feature space, and then calculating the projection length in the same direction to evaluate their similarity.

[0113] 3) Weighted fusion: According to the calculated attention weight, the outputs of CNN and GRU are weighted and fused. According to the attention weight, the output vector of CNN and the output matrix of GRU are weighted and summed to obtain a fused feature vector.

[0114] The fused feature vector will be passed to subsequent layers such as fully connected layer, Dropout, etc. for anomaly detection.

[0115] Based on the above description, compared with the prior art, the invention has the following advantages:

[0116] 1、The invention provides efficient and accurate log parsing function, which can process complex log data and extract useful information and key features. The invention improves the commonly used log parsing algorithm, and through the use of the log parsing method of the invention, users can automatically process and analyze large-scale log data. The log parsing function of the invention can accurately extract important information in the log, and through the extraction of the log template, users can better understand the organization structure and content of the log data, so that users can better understand the running state and behavior of the application system, and provide a basis for subsequent anomaly detection.

[0117] 2. The present application provides a solution to the problem of log data imbalance. In log anomaly detection, abnormal logs are often rare, while normal logs account for most of the data, leading to data imbalance. To overcome this problem, the present application introduces a negative sampling method. Through this method, the present application can adopt appropriate sampling strategies during training to balance the distribution of normal and abnormal samples. Through data enhancement, the algorithm of the present application can better cope with the challenges brought by data imbalance, and improve the performance and stability of the anomaly detection model.

[0118] 3. The present application constructs an efficient and accurate anomaly detection model based on convolutional neural network (CNN) and gated recurrent unit (GRU). In log anomaly detection, convolutional neural network (CNN) has excellent feature extraction ability and can automatically learn important local patterns and features from text data. Through the use of CNN, the anomaly detection model of the present application can effectively capture key information in log data, such as patterns and context relevance of abnormal events. CNN can extract local features through convolution operation and reduce feature dimension through pooling layer to retain important features. Gated recurrent unit (GRU) is a recurrent neural network structure suitable for sequence modeling, which can handle time series data and long-term dependencies. In log data, there is often a temporal correlation between events, and GRU can capture this temporal dependency. Through the use of GRU, the anomaly detection model of the present application can model the time series information of log data, thereby better understanding the context relationship between events and the evolution of abnormal behavior. By combining CNN and GRU, the anomaly detection model of the present application fully utilizes their advantages in log analysis. CNN can extract local features of log data, while GRU can capture temporal correlation. This combination can effectively identify and classify abnormal log events and improve the performance and accuracy of the anomaly detection model.

[0119] Further, the present application also provides the following implementation structure:

[0120] A log anomaly detection system applied to the log anomaly detection method provided above. The system comprises:

[0121] A preprocessing module for obtaining original log data and preprocessing the original log data to obtain log data. The original log data includes timestamp, log level and log content.

[0122] A log template analysis module for parsing log data using a fixed-depth tree-based log parsing algorithm to obtain log event templates.

[0123] The log sequence generation module is configured to construct a triple dictionary according to a log event template, and to perform negative sampling on logs in log data based on the triple dictionary to obtain a log sequence.

[0124] The vector conversion module is configured to convert the log sequence into a log sequence embedding vector based on a pre-trained language model.

[0125] The anomaly detection module is configured to construct an anomaly detection classifier based on a convolutional neural network and a gated recurrent unit, and to perform anomaly detection on the log sequence embedding vector.

[0126] An electronic device comprises:

[0127] A memory is configured to store a computer program.

[0128] A processor is connected to the memory and is configured to call and execute the computer program to implement the log anomaly detection method provided above.

[0129] In addition, the computer program in the memory described above is implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0130] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0131] The principles and implementation manners of the present application are described by using specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A log anomaly detection method, characterized by, The method comprises: obtaining original log data and preprocessing the original log data to obtain log data; the original log data comprises: a timestamp, a log level and log content; a log parsing algorithm based on a fixed-depth tree is used to parse the log data to obtain a log event template; the log parsing algorithm based on the fixed-depth tree comprises: setting a parameter variable, text preprocessing, searching according to log length, searching according to a constant token and similarity calculation; wherein the similarity calculation: the log is traversed from the first word to the last word or reaches a preset maximum tree depth, and the traversal ends; when the log reaches the last log event template layer, the similarity of the log and an existing template is calculated to determine the final log event template; if the similarity is greater than a threshold value, the log is considered to also have the log template; if the similarity is less than or equal to the threshold value, the Cartesian distance of the minhash of all log groups within a certain range before and after the log is compared to determine whether the Cartesian distance reaches a threshold value; if both similarity determination fails, a new log group is created, at which time complete placeholder replacement of the existing log is required, and finally the created new log category is added to the prefix search tree; a triple dictionary is constructed according to the log event template, and negative sampling is performed on the log in the log data based on the triple dictionary to obtain a log sequence, comprising: counting the triple of the log event appearing in all logs in the log data to obtain the triple frequency; based on the triple frequency, the triple of the log event in the log is replaced to complete the negative sampling; based on a pre-trained language model, the log sequence is converted into a log sequence embedding vector; an abnormality detection classifier based on a convolutional neural network and a gated recurrent unit is constructed to perform abnormality detection on the log sequence embedding vector.

2. The log anomaly detection method of claim 1, wherein, The preprocessing comprises: word segmentation processing and case conversion processing.

3. The log anomaly detection method of claim 1, wherein, The pre-trained language model is a word2vec model.

4. A log anomaly detection system, characterized by, The method is applied to the log abnormality detection method according to any one of claims 1-3; the system comprises: a preprocessing module configured to obtain original log data and preprocess the original log data to obtain log data; the original log data comprises: a timestamp, a log level and log content; a log template parsing module configured to parse the log data using a log parsing algorithm based on a fixed-depth tree to obtain a log event template; a log sequence generation module configured to construct a triple dictionary according to the log event template and perform negative sampling on the log in the log data based on the triple dictionary to obtain a log sequence; a vector conversion module configured to convert the log sequence into a log sequence embedding vector based on a pre-trained language model; an abnormality detection module configured to construct an abnormality detection classifier based on a convolutional neural network and a gated recurrent unit to perform abnormality detection on the log sequence embedding vector.

5. An electronic device, comprising: The system comprises: a memory configured to store a computer program; A processor, connected with the memory, is used to call and execute the computer program to implement the log anomaly detection method according to any one of claims 1-3.

6. The electronic device of claim 5, wherein, The memory is a computer readable storage medium.

Images