Log anomaly detection method based on bidirectional parallel tree optimization log parsing
Through the log analysis method based on bidirectional parallel tree and the LogBERT model, combined with the Transformer encoder, the problem of difficult time series and statistical features in log exception detection is solved, and the detection accuracy and accuracy are achieved, which is suitable for a variety of systems. .
Patent Information
- Application Number
- CN202410932859.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-12
AI Technical Summary
Existing log exception detection models are difficult to effectively capture the timing characteristics and statistical characteristics in log text, resulting in insufficient detection capabilities. The traditional model is only suitable for specific systems and ignores the context information of the log.
Using a log analysis method based on a bidirectional parallel tree, the log template is generated and abnormal sequence detection is performed through word segmentation, common variable filtering, bidirectional parallel tree construction, TF-IDF algorithm and Bi-kmeans clustering. Use the spherical objective function to adjust the distribution of normal and abnormal logs.
It improves the accuracy of log parsing and the accuracy of abnormal detection, can better capture the information of log sequences, and improves detection capabilities, especially on HDFS, BGL and Thunderbird datasets.
Smart Images

Figure CN118467743B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of log anomaly detection, and in particular to a log anomaly detection method based on bidirectional parallel tree optimized log parsing. Background Art
[0002] Online computer systems are vulnerable to various malicious attacks in cyberspace. Timely detection of abnormal events in online computer systems is a fundamental step in protecting the system. System logs record detailed information about computing events generated by computer systems and play an important role in today's anomaly detection. As data that records the real-time operating status of the system and the execution status of the program, logs are not only an important resource for software developers and operation and maintenance personnel to monitor the operating status of the system, but also one of the excellent data sources for realizing system anomaly detection. At present, log anomaly detection based on deep learning has become one of the key focuses in the field of anomaly detection in automated systems, but there are still the following problems that need to be solved urgently: the information and features contained in log text data are difficult to be directly learned by deep learning models as input, and specific log parsing methods need to be constructed; traditional log anomaly detection models are only suitable for specific systems, ignoring the unique temporal and statistical features of logs, making it difficult to capture the contextual information between events before and after the log, which leads to insufficient detection capabilities. Summary of the invention
[0003] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a log anomaly detection method based on bidirectional parallel tree optimization log parsing to improve the accuracy of log parsing, thereby improving the precision of log anomaly detection.
[0004] The technical solution adopted by the present invention is: a log anomaly detection method based on bidirectional parallel tree optimization log parsing, comprising the following steps:
[0005] Step S1, perform word segmentation and common variable filtering on the log data to obtain all word combinations of each log;
[0006] Step S2, creating an initial group by selecting the longest word combination from the word combinations whose frequencies are greater than the frequency threshold, and for each initial group, creating a bidirectional parallel tree with the longest common pattern as the root;
[0007] Step S3, updating the nodes in the parent direction of the bidirectional parallel tree, checking whether there are missing constant words in the column representing a frequency higher than the root node, and adding all the words in the column representing a frequency higher than the root node in the log message to the parent direction of the tree;
[0008] Step S4, updating nodes in the child direction of the bidirectional parallel tree, checking whether there are any missing constant words in the column with a lower representative frequency than the root node, and adding all words in the column with a lower representative frequency than the root node in the child direction of the tree;
[0009] Step S5: for each initial group, after the nodes in the parent direction and the child direction are added, the bidirectional parallel tree outputs a log template; each word in the initial group can correspond to a node in the tree, and the node is marked as a constant or a variable;
[0010] Step S6, using the TF-IDF algorithm to convert the text data in the log template into a feature vector; then using Bi-kmeans to cluster the log template, clustering the log template based on the feature vector obtained by the TF-IDF algorithm in the parent direction to identify the patterns that frequently appear in the log data; in the child direction, clustering is performed according to the structural features of the log template; through Bi-kmeans clustering, similar log templates are grouped to form different categories;
[0011] Step S7: Select entries that can represent log behaviors according to the log template, and the entries arranged in chronological order constitute a log sequence. Convert each entry in the log sequence into a log key, and the log keys are arranged in the order in the original log sequence to form a log key sequence; provide a unique event id for each parsed log key, and the event id is used as the vocabulary for training the LogBERT model; use a log sequence as the input data set of the LogBERT model, use a label to indicate whether the log sequence is normal or abnormal, and create a randomly generated matrix E∈R K×d , where each row in the matrix represents the embedding of each log key in the vocabulary, R is the dimension symbol, d is the dimension of each log key embedding, K represents a set of log keys extracted from the log message, creating a position embedding, and the position embedding D∈R K×d is generated by a sine function to encode the position information of the log key in the log sequence; the LogBERT model converts the log key k at time point t into t Represented as input representation x t , the input represents x t is the sum of log key embedding and position embedding, t∈1,2,…,T; T is the time period of the log sequence;
[0012] Step S8: The calculated input representation {x dist ,x 1 ,x 2 ,…,x t ,…,x T-1 ,x T} is fed as input to the Transformer encoder of the logBERT model; x dist is the input representation of the distance token added to the beginning of each log sequence, and the output of the Transformer encoder is the contextual embedding {h dist ,h 1,h 2 ,…,h t ,…,h T-1 ,h T}, corresponding to each log key in the log sequence, h dist is the context embedding vector of the distance sign, h t is the context embedding of the log key at time point t; embed the context of the masked mark h mask The masked token is passed to the fully connected layer and then to the softmax function to obtain the probability distribution of the tokens over the vocabulary; this probability distribution is used to predict the most appropriate masked token to replace the original masked token;
[0013] Step S9: Use a spherical objective function to adjust the distribution of normal log sequences so that normal log sequences are concentrated and close to each other in the embedding space, while abnormal log sequences are far away from the center of the sphere;
[0014] Step S10: After the training is completed, abnormal log sequence detection is performed.
[0015] Furthermore, in step S1, the logs are split into words, the frequency of each word in all logs is collected, a tuple is formed for each word, and tuples with the same frequency in each log are combined into word combinations.
[0016] Furthermore, the common variable filtering in step S1: first, a set of regular expression rules needs to be defined, and then the log data is read, each log is segmented, the continuous text is split into separate words or phrases, the segmented logs are iterated, and the defined regular expression rules are used to match the common variables; once the regular expression matches the common variable, the matched part is replaced with the wildcard "<*>", and after filtering out the common variables, the frequency of occurrence of each word in the entire log set is collected; a tuple is created for each word, including the word itself and its frequency, and word tuples with the same frequency are merged into a word combination.
[0017] Furthermore, in step S2, first, logs of the same length are grouped, and word combinations generated by each log with a frequency greater than a set frequency threshold are collected; secondly, each log selects the word combination with the most words as the longest word combination; finally, logs with the same longest word combination will be grouped; after the initial log group is created, the common longest word combination is the longest common pattern in each initial group.
[0018] Furthermore, in step S2, for each initial group, a bidirectional parallel tree with the longest common pattern as the root is created; words in the same column will be located at the same depth, and the maximum frequency of a word appearing in a column is considered to be the representative frequency of the column; the two directions of the bidirectional parallel tree are called the parent direction and the child direction, wherein the parent direction is used to check the missing of constant words in columns with a higher representative frequency than the root node, and the child direction is used to check the missing of constant words in columns with a lower representative frequency than the root node.
[0019] Further, in step S3, all words in the columns of the log message that represent a higher frequency than the root node will be added to the parent direction of the tree; if there are variable words, the number of different words in each parent column is calculated; if there are different words, the words in the column will be added to the tree as variable nodes; if there are multiple columns that meet the requirements of the parent node, the nodes will be added and classified in parallel.
[0020] Further, in step S4, the columns are sorted according to the number of different words, and the threshold of the number of different words in each column is used for classification; after the sorting is completed, the nodes in the sub-direction are added in order; if the number of different words in a column exceeds a preset threshold set according to empirical knowledge, the words in the column are classified as variable words; if the number of different words in the column does not exceed the preset threshold, the words in the column are classified as constants; once a new constant word is added to the bidirectional parallel tree, all logs containing the word from the root node to this leaf node will generate a new log group, and the subsequently added child nodes will be based on this new log group; this hierarchical process will be executed until the grouping is accurate and all constant words are found.
[0021] Furthermore, in step S7, each log sequence is now a sequence of log keys {k 1 , k 2 ,…, k t …, kᵀ - ¹, kᵀ}, kᵗ is the log key at time point t, k∈1,2,…,T-1,T; a unique distance marker k is added at the beginning of each log sequence dist .
[0022] Furthermore, in step S9, the spherical objective function formula is as follows:
[0023] ;
[0024] Among them, L VHM represents the spherical objective function loss value, is the context embedding vector of the distance marker, N is the number of context embedding vectors, c is the center of the normal log sequence in the training dataset, c=Mean(h dist), Mean means calculating the mean.
[0025] Further, in step S10, given a log sequence, firstly, a certain ratio of log keys are randomly replaced with mask tokens, and the randomly masked log sequence is used as the input of the LogBERT model; then, given a mask token, a candidate set consisting of g normal log keys is established; if the real log key is in the candidate set, the log key is regarded as a normal key; if the observed log sequence is not in the top-g candidate set predicted by the LogBERT model, the log sequence is considered to be an abnormal log sequence, and the top-g candidate set refers to the top g elements with the highest probability in a given probability distribution; then, when a log sequence consists of more than r abnormal log keys, the log sequence is marked as abnormal; g and r are both hyperparameters; when the LogBERT model predicts a masked log keyword, a probability distribution is generated.
[0026] The beneficial effects of the present invention are as follows: the parsing method based on the bidirectional parallel tree creates an initial group according to the longest common pattern, and then uses the bidirectional parallel tree to hierarchically supplement the constant words to the longest common pattern, thereby efficiently forming a complete log template. The method has achieved high parsing accuracy on three data sets: HDFS, BGL, and Thunderbird. LogBERT is a context-based model that can generate word embeddings based on the context, and can better capture the information of the entire log sequence, thereby improving the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 The figure is a flow chart of the method of the present invention.
[0028] Figure 2 Update the diagram for nodes in the parent direction.
[0029] Figure 3 It is the process of constructing the sub-direction tree. DETAILED DESCRIPTION
[0030] The present invention will be further described in detail below with reference to the accompanying drawings.
[0031] Reference Figure 1 , a log anomaly detection method based on bidirectional parallel tree optimization log parsing includes steps S1 to S10.
[0032] Step S1, log data preprocessing: perform word segmentation and common variable filtering on the log data to obtain all word combinations of each log.
[0033] In step S1, the log is split into words, the frequency of each word in all logs is collected, a tuple is formed for each word, and tuples with the same frequency in each log are combined into word combinations. Word segmentation process: L->W->T->F->C; L (log), W (word), T (tuple), F (frequency), C (word combination).
[0034] Common variable filtering in step S1: First, a set of regular expression rules need to be defined, which can identify common variables in the log. For example, a common regular expression for matching IP addresses may be \b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b. Then read the log data, perform word segmentation on each log, and split the continuous text into separate words or phrases. Iterate the word-segmented logs and use the defined regular expression rules to match possible common variables. Once the regular expression matches a common variable, replace the matched part with the wildcard "<*>". The purpose of this is to remove the specific values of these variables because they are irrelevant for analysis. After filtering out the common variables, collect the frequency of occurrence of each word in the entire log set. Create a tuple for each word, containing the word itself and its frequency. Merge word tuples with the same frequency into a word combination. Finally, output all the formed word combinations, which can be used for further data analysis or pattern recognition.
[0035] Step S2, initial group creation: create an initial group by selecting the longest word combination from the word combinations whose frequency is greater than the frequency threshold, and for each initial group, create a bidirectional parallel tree with the longest common pattern as the root.
[0036] Since constant words have more consistent frequencies than variable words, the longest word combination in each log is most likely to be part of the log template. However, when the frequencies of some variables are low, their frequencies may not interfere with each other, so that their frequencies are consistent and combined into the longest word combination. This situation results in only two word combinations in the log (i.e., variable and constant word combinations), and the variable word combination has more words than the constant word combination. So it is necessary to find these logs with only two different frequencies and check whether the longest word combination is a variable combination. If it is a variable combination, set a frequency threshold to avoid selecting these variable words. Therefore, step S2 of this embodiment creates an initial group by selecting the longest word combination from the word combination whose frequency is greater than the frequency threshold. The frequency threshold is set by multiplying the highest frequency of the word in the log by the hyperparameter weight.
[0037] In step S2, first, logs of the same length are grouped, and word combinations generated by each log with a frequency greater than a set frequency threshold are collected. Secondly, each log will select the word combination with the most words as the longest word combination. Finally, logs with the same longest word combination will be grouped. After the initial log group is created, the common longest word combination is the longest common pattern in each initial group.
[0038] In step S2, for each initial group, a bidirectional parallel tree with the longest common pattern as the root is created. Words in the same column will be at the same depth because they have the same classification. The maximum frequency of a word in a column is considered to be the representative frequency of the column. The two directions of the bidirectional parallel tree are called the parent direction and the child direction, where the parent direction is used to check the missing of constant words in columns with higher representative frequencies than the root node, and the child direction is used to check the missing of constant words in columns with lower representative frequencies than the root node.
[0039] Step S3, update the parent direction node: Figure 2 As shown, Figure 2 The hollow circles are constant nodes, and the solid circles are variable nodes. The node update in the parent direction of the bidirectional parallel tree checks whether there are missing constant words in the columns representing frequencies higher than the root node. All words in the columns of the log message representing frequencies higher than the root node will be added to the parent direction of the tree.
[0040] In step S3, all words in the columns of the log message that represent a higher frequency than the root node will be added to the parent direction of the tree. In these columns, if there are variable words, the probability of different words appearing is high, because variable positions always produce different words. Therefore, the number of different words in each parent column will be calculated. If there are different words, the words in that column will be added to the tree as variable nodes. If there are multiple columns that meet the needs of parent nodes, nodes will be added and classified in parallel.
[0041] Step S4, update sub-direction nodes: update the nodes in the sub-direction of the bidirectional parallel tree, check whether there are any missing constant words in the column with a lower representative frequency than the root node, and add all the words in the column with a lower representative frequency than the root node in the sub-direction of the tree.
[0042] In step S4, the words in the sub-columns with different words may also be constant words, so the rules of the parent direction cannot be used in this direction. Since variable positions produce more different words than fixed positions, the columns will be sorted according to the number of different words therein, and the threshold of the number of different words in each column will be used for classification. After the sorting is completed, the addition of nodes in the sub-direction can be performed in sequence. If the number of different words in a column exceeds the preset threshold set based on empirical knowledge, the words in the column are classified as variable words. If the number of different words in the column does not exceed the preset threshold, the words in the column are classified as constants. Once a new constant word is added to the bidirectional parallel tree, all logs containing words from the root node to this leaf node will generate a new log group, and the child nodes added subsequently will be based on this new log group. This hierarchical process will be performed until the grouping is accurate and all constant words are found.
[0043] Figure 3 This is the process of constructing the sub-direction tree. Figure 3 From the logs in, there are two columns with lower representative frequencies than the root node, namely the 4th and 5th columns. The words in the 5th column will be added to the tree first because it has a higher representative frequency. Two new constant nodes "HTTPS" and "SOCKS5" will be added to the root node because the number of branches is below the threshold. Then, the initial group is divided into two subgroups, namely subgroup 1 and subgroup 2, according to the left and right paths. The words in the fourth column of subgroup 1 will be added as child nodes of the node "HTTPS", and the words in the fourth column of subgroup 2 will be added as child nodes of the node "SOCKS5". Since the number of branches of these two nodes is greater than the threshold, these new nodes are classified as variable nodes.
[0044] Step S5, template generation: for each initial group, after the nodes in the parent direction and the child direction are added, the bidirectional parallel tree outputs the log template; each word in the initial group can correspond to a node in the tree, which is marked as a constant or a variable;
[0045] Step S6, Bi-kmeans clustering: Use the TF-IDF algorithm to convert the text data in the log template into a feature vector; then use Bi-kmeans to cluster the log template, cluster the log template in the parent direction based on the feature vector obtained by the TF-IDF algorithm to identify the patterns that frequently appear in the log data; in the child direction, cluster the log template according to the structural characteristics; through Bi-kmeans clustering, group similar log templates into different categories.
[0046] In step S6, the TF-IDF algorithm evaluates the importance of a word to a log template by calculating the frequency (TF) of each word in the log template and the inverse document frequency (IDF) in the entire log data set. TF-IDF converts log data into feature vectors to quantify the content of the log template.
[0047] Step S7: Feature extraction. According to the log template, select entries that can represent log behaviors, such as event type, timestamp, log level, etc. A series of entries arranged in chronological order constitute a log sequence. Each entry in the log sequence is converted into a log key. The log keys are arranged in the order in the original log sequence to form a log key sequence. The log key is a string template extracted from the log message by the log parser.
[0048] Provide a unique event id for each parsed log key, which is used as the vocabulary for BERT model training; when creating this vocabulary, only receive those events (log keys) whose occurrences exceed the specified threshold; convert log lines into log sequences.
[0049] Converting log lines into log sequences can be done in several different ways:
[0050] A. Sliding window: Use a sliding window with a window size to create a log sequence, where all logs in a time window (e.g., 5 minutes) are structured as a log sequence;
[0051] B. Fixed time window;
[0052] C. Methods based on log attributes;
[0053] D. Log line number method.
[0054] Some log sequences generated using a sliding window are shown below.
[0055]
[0056] As shown in the table above, each log sequence is now a sequence of log keys {k 1 , k 2 ,…, k t …, kᵀ - ¹, kᵀ}, kᵗ is the log key at time point t, k∈1,2,…,T-1,T; T is the time period of the log sequence. A unique distance marker k is added at the beginning of each log sequence dist The distance tag is used to compute the distance between this log sequence and the center (the center is computed using all log sequences in the input).
[0057] Take a log sequence as the input dataset, where the label indicates whether the log sequence is normal (0) or abnormal (1). Next, create a randomly generated matrix E ∈ R K×d , which represents log key embeddings (like word embeddings), where each row in the matrix represents the embedding of each log key in the vocabulary. R is the dimension symbol and d is the dimension of each log key embedding. K represents the set of log keys extracted from the log messages. In addition to this embedding, a position embedding is also created, and the position embedding D∈R K×d is generated by a sine function to encode the position information of the log key in the log sequence.
[0058] The LogBERT model converts the log key k at time point t t Represented as input representation x t , the input represents x t is the sum of the log key embedding and the position embedding, t∈1,2,…,T.
[0059] Step S8: Shield the log key prediction. dist , x 1 , x 2 , …,x t ,…, x T-1 ,x T} is fed as input to the Transformer encoder of the logBERT model. dist is the input representation of the distance token added to the beginning of each log sequence. The input representation is passed through a Transformer self-attention mechanism with stacked transformer layers similar to the BERT model; when the input representation is fed to the Transformer encoder, a certain fraction of the mask tokens are randomly masked as part of the training objective. The output of this Transformer encoder is the contextual embedding {h dist ,h 1 ,h 2 , …, h t ,…,h T-1 ,h T}, corresponding to each log key in the log sequence, h dist is the context embedding vector of the distance sign, h t is the context embedding of the log key at time t.
[0060] Embed the context of the masked token into h mask The masked tokens are passed to the fully connected layer and then to the softmax function to obtain the probability distribution of the tokens over the vocabulary; this probability distribution is used to predict the most appropriate masked token to replace the original masked token. The goal is to learn to predict the masked tokens in the log sequence, where the predicted probability distribution yʲmask The higher the probability of the actual log key in the MLKP The lower.
[0061] Step S9, minimize the volume of the hypersphere. A spherical objective function is used to adjust the distribution of normal log sequences so that normal log sequences should be concentrated and close to each other in the embedding space, while abnormal log sequences are far away from the center of the sphere. During the training phase, the LogBERT model learns / adjusts its weights to minimize the above loss function. The LogBERT model is trained on normal log sequences through the above two tasks, learning the patterns of normal log sequences to understand normal log patterns. Higher prediction accuracy can be achieved when predicting mask tokens in log sequences. The spherical objective function formula is as follows:
[0062] ;
[0063] Among them, L VHM represents the spherical objective function loss value, is the context embedding vector of the distance marker, N is the number of context embedding vectors, c is the center of the normal log sequence in the training dataset, c=Mean(h dist ), Mean means calculating the mean.
[0064] Step S10: After the LogBERT model training is completed, abnormal log sequence detection is performed. Given a test log sequence, first randomly replace a certain ratio of log keys with mask tokens (MASK), and use the randomly masked log sequence as the input of the LogBERT model; then, given a mask token, establish a candidate set consisting of g normal log keys; if the real log key is in the candidate set, the log key is regarded as a normal key; if the observed log sequence is not in the top-g candidate set predicted by the LogBERT model, the log sequence is considered to be an abnormal log sequence, and the top-g candidate set refers to the top g elements with the highest probability in a given probability distribution. Then, when a log sequence consists of more than r abnormal log keys, the log sequence is marked as abnormal. Both g and r are hyperparameters that can be adjusted according to the validation set; when the LogBERT model predicts a masked log keyword, a probability distribution is generated to indicate the probability of each possible keyword appearing in that position.
[0065] The specific implementation scheme described above further illustrates the purpose, technical scheme and technical effect of the present invention in detail. It should be understood that the above description is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the ideas and principles of the present invention should fall within the scope of protection of the present invention.
Claims
1. A log anomaly detection method based on bidirectional parallel tree optimized log parsing, characterized in that: The following steps are involved: Step S1, perform word segmentation and common variable filtering on the log data to obtain all word combinations of each log; the common variable filtering: first, a set of regular expression rules needs to be defined, and then the log data is read, and word segmentation is performed on each log, the continuous text is split into separate words or phrases, the logs after word segmentation are iterated, and the defined regular expression rules are used to match the common variables; once the regular expression matches the common variable, the matched part is replaced with the wildcard "<*>", and after filtering out the common variables, the frequency of occurrence of each word in the entire log set is collected; a tuple is created for each word, including the word itself and its frequency, and word tuples with the same frequency are merged into a word combination; Step S2, creating an initial group by selecting the longest word combination from the word combinations whose frequencies are greater than the frequency threshold, and for each initial group, creating a bidirectional parallel tree with the longest common pattern as the root; Step S3, updating the nodes in the parent direction of the bidirectional parallel tree, checking whether there are missing constant words in the column representing a frequency higher than the root node, and adding all the words in the column representing a frequency higher than the root node in the log message to the parent direction of the tree; Step S4, updating nodes in the child direction of the bidirectional parallel tree, checking whether there are any missing constant words in the column with a lower representative frequency than the root node, and adding all words in the column with a lower representative frequency than the root node in the child direction of the tree; Step S5: For each initial group, after the nodes in the parent direction and the child direction are added, the bidirectional parallel tree outputs a log template; each word in the initial group corresponds to a node in the tree, and the node is marked as a constant or a variable; Step S6: Use the TF-IDF algorithm to convert the text data in the log template into a feature vector; then use Bi-kmeans to cluster the log template, cluster the log template in the parent direction based on the feature vector obtained by the TF-IDF algorithm, and identify the patterns that frequently appear in the log data; In the sub-direction, clustering is performed based on the structural features of the log templates; Through Bi-kmeans clustering, similar log templates are grouped into different categories; Step S7: select items that can represent log behaviors according to the log template, and the items arranged in chronological order form a log sequence. Convert each item in the log sequence into a log key, and arrange the log keys in the order in the original log sequence to form a log key sequence; provide a unique event id for each parsed log key, and the event id is used as a vocabulary for LogBERT model training; Take a log sequence as the input dataset of the LogBERT model, indicated by the label Log sequence Is it normal or abnormal? Create a randomly generated matrix E ∈ R K×d , where each row in the matrix represents the embedding of each log key in the vocabulary, R is the dimension symbol, d Every Log key embedding The dimension of K represents a set of log keys extracted from log messages to create position embeddings, and the position embedding D∈ R K×d is generated by a sine function to log sequence middle Log Key Encode the location information; The LogBERT model converts the log of time point t into key k t Expressed as Input representation x t , lose Enter the expression x t It is a log key Embed and Positional Embedding The sum of , t∈{1,2,…,T}; T is Log sequence Time period; Step S8: The calculated input representation { x dist , x 1 , x 2 , …, x t ,…, x T-1 ,x T } As enter Transformer encoder fed to the logBERT model; x dist is added to each log sequence At the beginning Distance Marker The input representation of the Transformer encoder is Contextual Embedding { h dist , h 1 , h 2 , …, h t ,…,h T-1 ,h T }, right Should Log sequence Each log in key, h dist is the context embedding vector of the distance sign, h t is the context embedding of the log key at time t ; The mask mark Contextual Embedding mask Mask The token is passed to a fully connected layer and then to a softmax function to get the probability distribution of the token over the vocabulary; This probability distribution is used to predict the most appropriate mask token to replace the original mask token; Step S9, use A spherical objective function is used to adjust the distribution of normal log sequences so that normal log sequences are concentrated and close to each other in the embedding space, while abnormal log sequences are far away from the center of the sphere; Step S10: After the training is completed, abnormal log sequence detection is performed.
2. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 1 is characterized in that: In step S1, the logs are split into words, the frequency of each word in all logs is collected, a tuple is formed for each word, and tuples with the same frequency in each log will be combined into word combinations.
3. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 1 is characterized in that: In step S2, first, logs of the same length are grouped, and word combinations generated by each log with a frequency greater than a set frequency threshold are collected; secondly, the word combination with the most words is selected as the longest word combination for each log; finally, logs with the same longest word combination are grouped; after the initial log group is created, the common longest word combination is the longest common pattern in each initial group.
4. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 3 is characterized in that: In step S2, for each initial group, a bidirectional parallel tree with the longest common pattern as the root is created; words in the same column will be located at the same depth, and the maximum frequency of a word appearing in a column is considered to be the representative frequency of the column; the two directions of the bidirectional parallel tree are called the parent direction and the child direction, where the parent direction is used to check the missing of constant words in columns with higher representative frequencies than the root node, and the child direction is used to check the missing of constant words in columns with lower representative frequencies than the root node.
5. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 4 is characterized in that: In step S3, all words in the column of the log message that represent a higher frequency than the root node are added to the parent direction of the tree; if there are variable words, the number of different words in each parent column is calculated; if there are different words, the words in the column are added to the tree as variable nodes; If there are multiple columns that satisfy the requirements of the parent node, the nodes will be added and sorted in parallel.
6. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 5 is characterized in that: In step S4, the columns are sorted according to the number of different words, and the threshold of the number of different words in each column is used for classification; after the sorting is completed, the nodes in the sub-direction are added in order; if the number of different words in a column exceeds a preset threshold set according to empirical knowledge, the words in the column are classified as variable words; If the number of different words in the column does not exceed the preset threshold, the words in the column are classified as constants; once a new constant word is added to the bidirectional parallel tree, all logs containing the word from the root node to this leaf node will generate a new log group, and the subsequently added child nodes will be based on this new log group; This stratification process will continue until the grouping is precise and all constant words are found.
7. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 1 is characterized in that: In step S7, each log sequence is now a Sequence of log keys { k 1 , k 2 ,…, k t …, k ᵀ - ¹, k ᵀ },k ᵗ is the time point t Log Key , t∈{1,2,…,T-1,T}; a unique Distance Marker k dist .
8. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 7 is characterized in that: In step S9, the spherical objective function formula is as follows: ; Among them, L VHM represents the loss value of the spherical objective function, yes Distance Marker The context embedding vector is N, N is the number of context embedding vectors, and c is the normal Log sequence The center of dist ), Mean means calculating the mean.
9. The log anomaly detection method based on bidirectional parallel tree optimized log parsing according to claim 8 is characterized in that:In step S10, given a log sequence, firstly, a certain ratio of log keys are randomly replaced with mask tokens, and the randomly masked log sequence is used as the input of the LogBERT model; then, given a mask token, a candidate set consisting of g normal log keys is established; if the real log key is in the candidate set, the log key is regarded as a normal key; if the observed log sequence is not in the top-g candidate set predicted by the LogBERT model, the log sequence is considered to be an abnormal log sequence, and the top-g candidate set refers to the top g elements with the highest probability in a given probability distribution; then, when a log sequence consists of more than r abnormal log keys, the log sequence is marked as abnormal; g and r are both hyperparameters; when the LogBERT model predicts a masked log keyword, a probability distribution is generated.
