Log anomaly detection method based on Transform and time convolution

Abstract

The invention belongs to the field of log anomaly detection of a distributed system, and provides a log anomaly detection method based on Transform and time convolution, and the method comprises the following steps: S1, obtaining log data in a real environment; s2, log preprocessing: preprocessing the obtained log data set to obtain first log content; s3, dividing the log sequence: selecting a proper division strategy for the features of the first log content to obtain a second log content; s4, performing vectorization processing on the second log content to obtain a log vector fused with time, risk types and semantic features; s5, constructing a dual-channel anomaly detection model; and S6, traversing the log data set to train a model, and inputting the log vector into the trained anomaly detection model to perform log anomaly detection. According to the method, the problems of poor generalization ability and information loss in traditional log analysis are avoided, and the accuracy and efficiency of log anomaly detection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis and anomaly detection of distributed systems, and in particular to a log anomaly detection method based on Transformer and time convolution. Background Art

[0002] With the rapid development of cloud computing and distributed technology, the scale of software systems is becoming increasingly large, and the complexity is also increasing significantly, which makes the system more vulnerable to vulnerabilities and errors, thereby increasing the risk of system anomalies. The occurrence of anomalies may cause system service interruption, affect user experience, and even cause data damage or loss, resulting in significant economic losses. Therefore, how to effectively maintain the stability and reliability of software systems has become a vital task. In this context, it is particularly important to build an efficient and reliable anomaly detection model. The log files generated during the operation of the software system record the operating status of the system, including errors, anomalies, warnings and other information, and are often used by operation and maintenance personnel in troubleshooting, intrusion detection, performance analysis and other aspects, which also provides valuable data support for our anomaly detection.

[0003] Traditional manual analysis and rule-matching-based log anomaly detection methods rely on the professional knowledge of operation and maintenance personnel, consume large resources when facing large-scale logs, and often show problems of insufficient performance and low accuracy. Methods based on machine learning algorithms, such as principal component analysis (PCA) and support vector machine (SVM), have problems such as the need for manual feature engineering, difficulty in processing complex data, and low efficiency. This brings certain challenges to our use of logs for anomaly detection. In recent years, with the rapid development of deep learning technology, it has been widely used in various fields, bringing new interpretation ideas to log anomaly detection and showing excellent performance. Deep learning models can automatically learn abnormal features in log sequences, improve adaptability to new threats, reduce excessive reliance on professional knowledge, and effectively improve the effect of log anomaly detection.

[0004] At present, deep learning-based log anomaly detection methods usually include three steps: (1) log parsing, parsing logs into log templates and parameters; (2) constructing log template sequence count vector matrix or semantic vector matrix; (3) anomaly detection based on deep learning models. However, these aspects still face some challenges. First, the log parsing process is prone to information loss and may introduce noise, which affects the performance of the anomaly detection model. Second, the RNN model and its variants are commonly used in the current log anomaly detection framework, but these models have limitations in capturing long-distance dependencies, local features, and temporal dependencies in log sequences. In addition, although some frameworks have introduced the Transformer model to enhance feature extraction capabilities, the high computational complexity and increased number of parameters also limit work efficiency. Summary of the invention

[0005] In view of the limitations of existing log anomaly detection methods, the present invention provides a log anomaly detection method based on Transformer and temporal convolution to overcome the problems of information loss and poor generalization ability of existing log parsing methods, insufficient feature extraction of detection models, and high computational overhead. The present invention uses a dual-channel model to extract features, adopts a Transformer module to extract global features of log data, and uses a temporal convolutional network to capture local features in the sequence, fully mining the temporal features and semantic features of log data. At the same time, the present invention introduces a separable self-attention mechanism and a point-by-point convolution optimization model structure to further reduce computational complexity, thereby effectively improving the accuracy and work efficiency of anomaly detection.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A log anomaly detection method based on Transformer and time convolution, characterized by comprising the following steps:

[0008] Step S1: Obtain log data in a real environment;

[0009] Step S2: log preprocessing: preprocess the acquired log data set, parse the timestamp and risk type fields in the log header, and extract the original text data of the log message content to obtain the first log content; store the parsed first log content in a structured format for subsequent feature extraction and model training;

[0010] Step S3: Divide the log sequence: select a suitable division strategy based on the features of the first log content to obtain the second log content;

[0011] Step S4: vectorizing the second log content to obtain a log vector that integrates time, risk level, and semantic features;

[0012] Step S5: construct a dual-channel anomaly detection model: the main channel of the model is a single-layer Transformer encoder, and the side channel is a temporal convolutional network;

[0013] Step S6: traverse the log data set to train the model, and input the log vector into the trained anomaly detection model to perform log anomaly detection.

[0014] Furthermore, the log data of S1 is collected and generated by the open source platform LogHub in a real environment.

[0015] Furthermore, the step S3 is specifically as follows:

[0016] For the HDFS dataset, the log contains the key identifier block_id, and session window partitioning is used; for the BGL dataset, fixed window or sliding window partitioning is used; the original text of the log message content is segmented using regular expressions.

[0017] Furthermore, the step S4 of performing vectorization processing on the second log content is specifically as follows:

[0018] Time vector: Only month, day, hour, minute and second are used, denoted as , and the time data is normalized to [0,1], and the normalization method is as follows:

[0019]

[0020] The normalized time data is taken as the time vector, denoted as: V Time =(t' month ,t' day ,t' hour ,t' min ,t sec ').

[0021] Furthermore, the step S4 of performing vectorization processing on the second log content also includes:

[0022] Risk type vector: The Log4j log framework defines 8 log levels with different priorities in detail, which are OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL from high to low. Given that the risk levels in our log data are discrete data and there is a clear order relationship between the levels, we choose to use the label encoding method for vectorization, mapping the log risk levels from low to high to integers from 0 to 7. The resulting risk type vector is recorded as: V Level .

[0023] Furthermore, the step S4 of performing vectorization processing on the second log content also includes:

[0024] Semantic vector: Log messages are regularized to remove special characters, punctuation and meaningless content, and the processed log messages are segmented into word lists L = [w 1 ,w 2 ,...w n ];

[0025] Use FastText to train each word w i Perform word embedding and generate word vector V i FT , the specific formula is:

[0026] V i FT =FastText(w i )

[0027] Use pre-trained BERT for each word w i Perform word embedding and generate corresponding word vectors The specific calculation formula is:

[0028] V i BT =BERT(w i )

[0029] Concatenate the word embedding vectors generated by FastText and BERT, and merge the embedding vectors of each word into a concatenated vector V i CT , the calculation formula is:

[0030] V i CT =[V i FT ; V i BT ]

[0031] For all concatenated vectors V in the log message content i CT Perform average pooling to generate the semantic feature vector V of the log message content l-sem , the calculation formula is:

[0032]

[0033] Where n is the number of word tokens in the log message content.

[0034] Furthermore, the Transformer encoder of the main channel single layer includes position embedding, separable self-attention mechanism, normalization layer and feed-forward neural network layer, which can effectively capture the long-distance dependency of the log sequence;

[0035] Position embedding: provides the model with the relative position relationship of each log in the log sequence, which helps capture the structure and semantic information of the log sequence. The specific calculation formula is as follows:

[0036]

[0037] Among them, pos is the position of the log in the sequence, d model is the dimension of the feature vector, i is the dimension information;

[0038] Separable self-attention mechanism: The calculation formula is as follows:

[0039]

[0040] Given input data X∈R K*d , input X into three branches, namely I, K, and V. The linear W in the I branch uses the weight 1 ∈R d The data obtained by layer processing is passed through Softmax as the context score C s ; K branches pass through weight W K ∈R d*d The linear layer is processed to obtain X K ∈R K*d , get the context variable C V , the calculation formula is as follows:

[0041]

[0042] The last branch V, input data X uses weight W V ∈R d*d After the linear layer processing and ReLU, the data X is obtained V ∈R K*d , the context variable C V With X V The result of element-wise multiplication is sent to a weight W O ∈R d*d The linear layer gets the final output Y∈R K*d ;

[0043] Among them, σ is Softmax.

[0044] Normalization layer 1: It includes residual connection and LayerNorm, which are specifically expressed as follows:

[0045] Z 1 =LayerNorm(X+Separable-Self-Attention(X))

[0046] Among them, X represents the input of the Separable Self-Attention layer;

[0047] Normalization layer 2: It includes residual connection and LayerNorm, which are specifically expressed as follows:

[0048] Z 2 =LayerNorm(Z 1 +FFN(Z 1 ))

[0049] Feedforward neural network layer: It consists of two linear mappings. The activation function ReLU is used after the first linear mapping. The calculation formula is as follows:

[0050] FFN(Z 1 )=ReLU(Z 1 W 1 +b 1 )W 2 +b 2

[0051] Where W 1 , W 2 and b 1 、b 2 Randomly initialize the weights and biases for the two linear maps respectively.

[0052] Furthermore, the side channel can capture features of different scales by setting the expansion rate. Four expansion rates of 1, 2, 4, and 8 are set in the model.

[0053] The features extracted from the main channel and the side channel are fused and input into a multi-layer perceptron (MLP) for classification.

[0054] Furthermore, the model training uses the Adamw optimizer and the cross-entropy loss function, and the cross-entropy loss function is defined as follows:

[0055]

[0056] Where N is the number of samples; y i is the sample label, which takes the value of 0 or 1, y i =1 indicates an abnormal log sequence, and 0 indicates a normal log sequence; p i It indicates the predicted probability that the sample belongs to an abnormal log sequence, and its value range is (0, 1).

[0057] The beneficial effects of the present invention are: avoiding the problems of poor generalization ability and information loss in traditional log parsing, making full use of log data characteristics, solving the problems of high computing resource requirements and difficulty in capturing local correlations of the Transformer model, thereby improving the accuracy and efficiency of log anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a flow chart of the log anomaly detection method provided by this application;

[0060] Figure 2 It is a log data processing diagram provided by this application;

[0061] Figure 3 It is a schematic diagram of the overall framework of the log anomaly detection method provided by this application;

[0062] Figure 4 It is a schematic diagram of the structure of the residual block in the temporal convolutional network provided by this application;

[0063] Figure 5 It is a structural diagram of the Transformer encoder provided in this application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Figure 1 A flow chart of a log anomaly detection method provided in an embodiment of the present application is provided. Figure 3 This is a schematic diagram of the overall framework of the log anomaly detection method provided by this application, such as Figure 1 , Figure 3 As shown, the present application provides a log anomaly detection method based on Transformer and time convolution, including the following specific steps:

[0066] Step S1: Obtain log data in a real environment;

[0067] It should be noted that the log data is collected and generated by the open source platform LogHub in a real environment.

[0068] Step S2: preprocessing of logs: after obtaining the log data set, preprocess the log data, including parsing the timestamp and risk type fields in the log header, and extracting the original text data of the log message content to obtain the first log content, and store the parsed first log content in a structured format for subsequent feature extraction and model training.

[0069] Step S3: Divide the log sequence: select a suitable division strategy based on the features of the first log content to obtain the second log content; the division strategy of the log sequence adopts different division strategies to obtain the log sequence according to the characteristics of the log data set collection.

[0070] It should be noted that for the HDFS data set, the log contains the key identifier block_id, and the session window partitioning method can be used to construct the sequence LogSeq = (l 1 ,l 2 ,l 3 ,...l n ), where n represents the length of the sequence. For the BGL dataset, due to the lack of such specific identifiers, a fixed window or sliding window method is used to divide the log sequence.

[0071] The original text of the log message content is segmented, and regular expressions are used to remove common variables such as IP addresses and block_ID based on domain knowledge, and special characters, punctuation marks and other meaningless content in the log are replaced with spaces. After a series of preprocessing steps, we divide the log message content into a set of word lists. For example, for the message content:

[0072] “dfs.DataNode$PacketResponder:Received block blk_-1608999687919862906of size 91178from / 10.250.14.224”, after processing, we will get the word list [dfs, data, node, packet, responder, received, block, of, size, from].

[0073] Step S4: vectorizing the second log content to obtain a log vector that integrates time, risk level, and semantic features;

[0074] It should be noted that if Figure 2As shown in the figure, the time vector, taking the HDFS log as an example, the timestamp format is YYMMDDHHMMSS, which is the last two digits of the year, month, day, hour, minute and second. Only the month, day, hour, minute and second are used, and t = (t month， t day， t hour， t min， t sec ). In order to eliminate the scale effect of time data, we normalize the time data to the range of [0,1]. The normalization method is as follows:

[0075]

[0076] Use the normalized time data as the time vector, denoted as: V Time =(t' month ,t' day ,t' hour ,t' min ,t sec ').

[0077] Risk type vector, the Log4j log framework defines 8 log levels of different priorities in detail, which are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL from high to low. Given that the risk levels in our log data are discrete data and there is a clear order relationship between the levels, it is most appropriate to use the label encoding method for vector representation. Map the log risk levels from low to high to integers from 0 to 7, and the resulting risk type vector is recorded as: V Level .

[0078] Semantic vector: Log messages are regularized, meaningless content such as special characters and punctuation marks are removed, and then word segmentation is performed to divide the log message content into a word list L = [w 1 ,w 2 ,...w n ].

[0079] Use FastText to train each word w i Perform word embedding and generate word vector V i FT , the specific formula is:

[0080] V i FT =FastText(w i )

[0081] Use pre-trained BERT for each word w i Perform word embedding and generate word vector V iBT , the specific formula is:

[0082] V i BT =BERT(w i )

[0083] The word embedding vectors of FastText and BERT are concatenated, and the embedding vectors of each word are merged into a comprehensive vector V i CT , the calculation formula is:

[0084] V i CT =[V i FT ; V i BT ]

[0085] For all concatenated vectors V in the log message content i CT Perform average pooling to generate the semantic feature vector V of the log message content l-sem , the calculation formula is:

[0086]

[0087] Where n is the number of word tokens in the log message content.

[0088] Step S5: construct a dual-channel anomaly detection model: the main channel of the model is a single-layer Transformer encoder, and the side channel is a temporal convolutional network;

[0089] It should be noted that the process of building a dual-channel anomaly detection model is as follows: by introducing the Transformer Encoder module to model the temporal relationship as the main channel, the global features or long-distance dependencies of the sequence can be fully captured; at the same time, the TCN module side channel is used as the dynamic modeling layer to effectively process the temporal dependencies in the log data, and the local features of the sequence can be captured through convolution operations to adapt to the dynamic changes of the system.

[0090] like Figure 5 As shown in Figure 1, a single-layer Transformer encoder in the main channel, including position embedding, separable self-attention mechanism, normalization layer and feed-forward neural network layer, can effectively capture long-distance dependencies.

[0091] Position embedding: provides the model with the relative position relationship of each log in the log sequence, which helps capture the structure and semantic information of the log sequence. The specific calculation formula is as follows:

[0092]

[0093] Among them, pos is the position of the log in the sequence, d model is the dimension of the feature vector, i is the dimension information;

[0094] Separable self-attention mechanism: The calculation formula is as follows:

[0095]

[0096] Given input data X∈R K*d , X is input into three branches, namely I, K, and V. The linear W in the I branch uses the weight 1 ∈R d The data obtained by layer processing is passed through Softmax as the context score C s ; K branches pass through weight W K ∈R d*d The linear layer is processed to obtain X K ∈R K*d , get the context variable C V , the calculation formula is as follows:

[0097]

[0098] The last branch V, input data X uses weight W V ∈R d*d After the linear layer processing and ReLU, the data X is obtained V ∈R K*d , the context variable C V With X V The result of element-wise multiplication is sent to a weight W O ∈R d*d The linear layer gets the final output Y∈R K*d ;

[0099] Among them, σ is Softmax;

[0100] Normalization layer 1: It includes residual connection and LayerNorm, which are specifically expressed as follows:

[0101] Z 1 =LayerNorm(X+Separable-Self-Attention(X))

[0102] Among them, X represents the input of the Separable Self-Attention layer;

[0103] Normalization layer 2: It includes residual connection and LayerNorm, which are specifically expressed as follows:

[0104] Z2 =LayerNorm(Z 1 +FFN(Z 1 ))

[0105] Feedforward neural network layer: It consists of two linear mappings. The activation function ReLU is used after the first linear mapping. The calculation formula is as follows:

[0106] FFN(Z 1 )=ReLU(Z 1 W 1 +b 1 )W 2 +b 2

[0107] Among them, W 1 , W 2 and b 1 , b 2 Randomly initialize the weights and biases for the two linear maps respectively.

[0108] like Figure 4 As shown in the figure, the side channel introduces a TCN model based on a convolutional structure, which can capture local features in log sequences through stacked convolutional layers. Its parallel operation can accelerate the feature extraction process, making up for the shortcomings of Transformer in capturing local features of log sequences. Its dilated convolution is the key to obtaining a flexible receptive field. By setting the dilation rate, we can capture features of different scales and extract features in a multi-scale range. Four dilation rates of 1, 2, 4, and 8 are set in the model. Finally, the features extracted by the main channel and the side channel are fused and input into a multi-layer perceptron (MLP) for classification.

[0109] Step S6: traverse the log data set to train the model, and input the log vector into the trained anomaly detection model to perform log anomaly detection.

[0110] It should be noted that during the model training process, the AdamW optimizer is used for gradient update, the initial learning rate is set to 0.001, and the learning rate is dynamically adjusted in combination with the cosine annealing algorithm. In order to improve the performance and convergence efficiency of the model, the cross-entropy loss function is used to calculate the classification error. The definition of the cross-entropy loss function is as follows:

[0111]

[0112] Where N is the number of samples; y i is the sample label, which takes the value of 0 or 1, y i =1 indicates an abnormal log sequence, and 0 indicates a normal log sequence; p iIt indicates the predicted probability that the sample belongs to an abnormal log sequence, and its value range is (0, 1). By minimizing the loss function, the predicted probability of the model tends to the distribution of the true label.

[0113] Other hyperparameters are set as follows: batch size is 128, training epochs is 30, and dropout probability is set to 0.1 to randomly discard some neuron outputs to prevent overfitting.

[0114] After each round of training, the validation set is used to evaluate the performance of the model and calculate the accuracy, F1 value, and validation set loss value. In each training cycle, if the validation set performance does not improve, the learning rate reduction strategy is used to try to improve the model performance.

[0115] Training termination conditions: stop training when the maximum number of epochs is reached, or terminate training early when the validation set loss value does not decrease for three consecutive epochs (Early Stopping) to avoid overfitting and save computing resources. Finally, save a set of model parameters with the best validation set performance for subsequent applications.

[0116] It should be noted that anomalies in anomaly detection are mainly divided into two categories: point anomalies and context anomalies. Point anomalies refer to anomalies in a single log entry, such as a log containing clear abnormal keywords or an error code. Context anomalies refer to a log entry that is normal in itself, but the context in which it appears does not meet expectations, such as an abnormal operation sequence in a log sequence or an abnormal task interval (a log shows that the request response time suddenly increases during non-busy hours).

[0117] Specific examples are as follows:

[0118] The HDFS and BGL datasets collected by the open source platform LogHub in a real environment were used. The HDFS dataset has a total of 11,175,629 logs. Professionals divided the log sequences according to the specific block_id in each log, and marked the log sequences according to domain knowledge, dividing them into normal and abnormal sequences. Among the 575,061 divided log sequences, there are 16,838 abnormal sequences, accounting for about 2.9% of the total sequences. The BGL dataset has a total of 4,747,963 log data. The logs were divided into two categories by professionals: alarm information and non-alarm information, of which alarm information can be regarded as abnormal. There are 348,460 abnormal logs in the dataset, accounting for about 7.3% of the total logs.

[0119] To evaluate its performance, it is compared with several classic deep learning methods, including DeepLog based on log template index, LogAnomaly based on log template semantics, and LightLog. Experiments are conducted on the public HDFS and BGL log datasets, and multiple evaluation indicators such as Precision, Recall, and F1-Score are used to evaluate the performance of the model.

[0120] From Table 1, we can see that the method based on log template index has relatively poor detection effect because it ignores the sequential relationship between logs, while the introduction of log template semantics significantly improves the accuracy of log anomaly detection. The accuracy of LogAnomaly and LightLog on the BGL dataset is improved by 4.7% and 2.9% respectively compared with the DeepLog method. However, there are also some problems based on LSTM and CNN network models. LSTM may cause gradient explosion or disappearance problems, and it is difficult to focus on the local features of the log sequence. Although CNN is good at processing local features, it may ignore global context information when processing long sequences. The method in this paper combines the advantages of TCN and Transformer. It can not only capture long-distance dependencies, but also effectively extract local features. It has better robustness and performs well compared with other methods in terms of Precision, Recall and F1-Score indicators.

[0121] Table 1: Experimental results on HDFS and BGL datasets

[0122]

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log anomaly detection method based on Transformer and time convolution, characterized in that: The following steps are involved: Step S1: Obtain log data in a real environment; Step S2: log preprocessing: preprocess the acquired log data set, parse the timestamp and risk type field in the log header, and extract the original text data of the log message content to obtain the first log content; Step S3: Divide the log sequence: select a suitable division strategy based on the features of the first log content to obtain the second log content; Step S4: vectorizing the second log content to obtain a log vector that integrates time, risk level, and semantic features; Step S5: construct a dual-channel anomaly detection model: the main channel of the model is a single-layer Transformer encoder, and the side channel is a temporal convolutional network; Step S6: traverse the log data set to train the model, and input the log vector into the trained anomaly detection model to perform log anomaly detection.

2. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The log data of S1 is collected and generated by the open source platform LogHub in a real environment.

3. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The step S3 is specifically as follows: For the HDFS dataset, the log contains the key identifier block_id, and session window partitioning is used; for the BGL dataset, fixed window or sliding window partitioning is used; the original text of the log message content is segmented using regular expressions.

4. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The step S4 of vectorizing the second log content is specifically as follows: Time vector: only month, day, hour, minute and second are used, denoted as t = (t month， t day， t hour， t min， t sec ), normalize the time data to [0,1], the normalization method is as follows: The normalized time data is taken as the time vector, denoted as: V Time =(t' month ,t' day ,t' hour ,t' min ,t sec ').

5. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The step S4 of vectorizing the second log content further includes: Risk type vector: The Log4j log framework defines 8 log levels with different priorities in detail, which are OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL from high to low. Given that the risk levels in our log data are discrete data and there is a clear order relationship between the levels, we choose to use the label encoding method for vectorization, mapping the log risk levels from low to high to integers from 0 to 7. The resulting risk type vector is recorded as: V Level .

6. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The step S4 of vectorizing the second log content further includes: Semantic vector: Log messages are regularized to remove special characters, punctuation and meaningless content, and the processed log messages are segmented into word lists L = [w1, w2, ... w n ]; Use FastText to train each word w i Perform word embedding and generate word vector V i FT , the specific formula is: V i FT =FastText(in i ) Use pre-trained BERT for each word w i Perform word embedding and generate the corresponding word vector V i BT , the specific calculation formula is: V i BT =BERT(w i ) Concatenate the word embedding vectors generated by FastText and BERT, and merge the embedding vectors of each word into a concatenated vector V i CT , the calculation formula is: V i CT =[V i FT ;V i BT ] For all concatenated vectors V in the log message content i CT Perform average pooling to generate the semantic feature vector V of the log message content l-sem , the calculation formula is: Where n is the number of word tokens in the log message content.

7. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The Transformer encoder of the main channel single layer includes position embedding, separable self-attention mechanism, normalization layer and feed-forward neural network layer, which can effectively capture the long-distance dependencies of log sequences; Position embedding: provides the model with the relative position relationship of each log in the log sequence, which helps capture the structure and semantic information of the log sequence. The specific calculation formula is as follows: Where pos is the position of the log in the sequence, d model is the dimension of the feature vector, i is the dimension information; Separable self-attention mechanism: The calculation formula is as follows: Given input data X∈R K*d , input X into three branches, namely I, K, and V, and use weight W1∈R in branch I d The linear layer processes the data and passes through Softmax as the context score C s ; K branches pass through weight W K ∈R d*d The linear layer is processed to obtain X K ∈R K*d , get the context variable C V , the calculation formula is as follows: The last branch V, input data X uses weight W V ∈R d*d After the linear layer processing and ReLU, the data X is obtained V ∈R K *d , the context variable C V With X V The result of element-wise multiplication is sent to a weight W O ∈R d*d The linear layer gets the final output Y∈R K*d ; Among them, σ is Softmax; Normalization layer 1: It includes residual connection and LayerNorm, which are specifically expressed as follows: Z1=LayerNorm(X+Separable-Self-Attention(X)) Among them, X represents the input of the Separable Self-Attention layer; Normalization layer 2: It includes residual connection and LayerNorm, which are specifically expressed as follows: Z2=LayerNorm(Z1+FFN(Z1)) Feedforward neural network layer: It consists of two linear mappings. The activation function ReLU is used after the first linear mapping. The calculation formula is as follows: FFN(Z1)=ReLU(Z1W1+b1)W2+b2 Among them, W1, W2 and b1, b2 are the randomly initialized weights and biases of the two linear mappings respectively.

8. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The side channel can capture features of different scales by setting the expansion rate. Four expansion rates of 1, 2, 4, and 8 are set in the model.

9. The log anomaly detection method based on Transformer and time convolution as claimed in claim 1, characterized in that: The model training uses the Adamw optimizer and the cross entropy loss function. The cross entropy loss function The definition of (Cross-Entropy Loss) is as follows: Where N is the number of samples; y i is the sample label, which takes the value of 0 or 1, y i =1 indicates an abnormal log sequence, and 0 indicates a normal log sequence; p i It indicates the predicted probability that the sample belongs to an abnormal log sequence, and its value range is (0, 1).

Images