Cross-domain model training and log anomaly detection method and device based on transfer learning

Abstract

This application provides a cross-domain model training method based on transfer learning, comprising the following steps: A1. Performing sliding window partitioning on the source system log messages and the target system log messages to obtain corresponding source system log sequences and target system log sequences; A2. Performing equal partitioning on the source system log sequences and the target system log sequences to obtain log sequence pairs; A3. Performing parsing and transformation processing on the source system log messages and the target system log messages to obtain log template vectors; A4. Training the model based on the log sequence pairs, the log template vectors, and the total loss function to obtain a trained LSTM model and a hypersphere model. The technical solution involved in this application, by employing a contrastive learning method to compare the similarity between two features in pairs, is beneficial for quantifying the differences between features, which can reduce model training costs and enhance the detection effect of log anomaly detection.

Description

Technical Field

[0001] This application relates to the field of log anomaly detection technology, and in particular to cross-domain model training and log anomaly detection methods, devices and storage media based on transfer learning. Background Technology

[0002] System logs record detailed operational information. Generally, the cause of a fault is also recorded in the system logs. Analyzing and detecting these logs can provide multi-dimensional information for fault localization. Log anomaly detection can help debug and analyze the root cause of system failures, providing reliable service. In production services, a newly deployed system, due to its short runtime, has a small total number of collected logs, making it impossible to train a detection model, resulting in a cold start problem for log anomaly detection. Transfer learning is an effective method to solve the cold start problem in log anomaly detection. Transfer learning transfers knowledge from one domain (source domain) to another domain (target domain), which can significantly improve log anomaly detection performance even with insufficient samples.

[0003] However, different manufacturers and models of equipment, as well as system logs, exhibit differences in format, syntax, and semantics, lacking a unified standard and resulting in inconsistent log specifications. When software systems perform different tasks, their component calls, I / O outputs, and fault types also differ. Therefore, based on the similarity of the service domains of the source and target systems, migrations can be categorized into two types: same-domain cross-system migration and cross-domain migration. Same-domain cross-system migration refers to the source and target systems having similar system service objects, differing only in log syntax and format. Cross-domain migration refers to the source and target systems having different system service objects and operational logic. For example, BGL (BlueGene / L supercomputer), HPC (high-performance cluster), and Thunderbird are all supercomputer systems; HDFS, Hadoop (WordCount, PageRank), and Spark are all distributed systems; Windows, Linux, and Mac are all operating systems. Cross-system migration is migration between systems within the same domain, such as Windows->Linux, BGL->Thunderbird. Cross-domain migration is migration between systems in different domains, such as Windows->Hadoop, BGL->Hadoop.

[0004] Existing transfer learning methods are all cross-system transfers, but in real-world environments, due to the lack of datasets, there is a greater need for cross-domain transfer learning. However, when the source system and target system are in different domains, the model's anomaly detection performance in the target system is poor.

[0005] Therefore, how to improve the detection effect of log anomaly detection under the condition of small sample size in the target system has become a problem that needs to be solved.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may contain information that is not part of the prior art known to those skilled in the art. Summary of the Invention

[0007] This application provides a model training and log anomaly detection method, device and storage medium to solve the problems existing in the prior art.

[0008] In a first aspect, this application provides a model training method, comprising the following steps: A1. Performing sliding window partitioning on source system log messages and target system log messages to obtain corresponding source system log sequences and target system log sequences; A2. Performing equal partitioning on the source system log sequences and the target system log sequences to obtain log sequence pairs, wherein each log sequence pair includes a first subsequence set and a second subsequence set, and both the first subsequence set and the second subsequence set include multiple log sequences; A3. Performing parsing and transformation processing on the source system log messages and the target system log messages to obtain log template vectors; A4. Performing model training based on the log sequence pairs, the log template vectors, and the total loss function to obtain a trained LSTM (Long Short-Term Memory) model and a hypersphere model; wherein the total loss function includes: a hypersphere loss function, an alignment loss function, and a uniform loss function, wherein the alignment loss function is used to align or shorten the distance between the same pair of log sequence features, and the uniform loss function is used to make the log sequence features evenly distributed on the hypersphere.

[0009] In some embodiments, the hyperspherical loss function Loss h for:

[0010]

[0011] Wherein, V1 represents the first log sequence feature set formed after the first subsequence set is extracted by the LSTM model, and V2 represents the second log sequence feature set formed after the second subsequence set is extracted by the LSTM model. i This represents the feature of a single log sequence, where C represents the hyperspherical center feature.

[0012] The total loss function Loss sum for:

[0013] Loss sum =α1*Loss h +α2*Loss align +Loss uniform

[0014] Where α1 and α2 are the hyperparameters balancing the three loss functions, and Loss align Represents the alignment loss function, Loss uniform Represents the uniform loss function, Loss h This represents the hyperspherical loss function.

[0015] In some embodiments, the alignment loss function Loss align for:

[0016]

[0017] in, This represents the i-th log sequence feature in the first log sequence feature set. N represents the i-th log sequence feature in the second log sequence feature set. sub This represents the total number of log sequence features in a single log sequence feature set.

[0018] In some embodiments, the uniform loss function Loss uniform for:

[0019]

[0020] Where e represents the base of the natural logarithm, This represents the i-th log sequence feature in the first log sequence feature set. N represents the i-th log sequence feature in the second log sequence feature set. sub This represents the total number of log sequence features in a single log sequence feature set.

[0021] In some embodiments, A2 includes: A21, mixing and randomly shuffling the source system log sequence and the target system log sequence to obtain a mixed log sequence; A22, dividing the mixed log sequence into two equal sets of sub-log sequences to obtain the log sequence pair.

[0022] In some embodiments, the model training method further includes: extracting the log sequence features based on the LSTM model, adjusting and determining the decision boundary, the decision boundary being used to distinguish between normal log sequence features and abnormal log sequence features, obtaining the decision boundary distance, the decision boundary distance being the distance from the decision boundary to the center of the hypersphere model.

[0023] Secondly, this application provides a log anomaly detection method, which is implemented using an LSTM model and a hypersphere model. The LSTM model and the hypersphere model are trained according to the model training method. The log anomaly detection method includes the following steps: B1, performing sliding window segmentation on the target system log messages to be detected to obtain a target system log sequence; B2, parsing the target system log sequence into a log template, and obtaining a log template vector based on the log template; B3, inputting the log template vector into the trained LSTM model to obtain a log sequence feature set, which includes multiple log sequence features; B4, inputting the log sequence features into the trained hypersphere model to obtain anomaly detection results.

[0024] In some embodiments, B4 includes: B41, calculating a first distance from the log sequence features to the center of the hypersphere model; B42, comparing the first distance with the decision boundary distance to obtain a comparison result; and B43, obtaining an anomaly detection result based on the comparison result.

[0025] Thirdly, this application provides a terminal device, including:

[0026] Memory, used to store computer programs;

[0027] A processor is configured to read a computer program from the memory and execute the cross-domain model training method based on transfer learning or the cross-domain log anomaly detection method based on transfer learning.

[0028] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the cross-domain model training method based on transfer learning or the cross-domain log anomaly detection method based on transfer learning.

[0029] The model training method provided in this application includes the following steps: A1. Performing sliding window partitioning on the source system log messages and the target system log messages to obtain corresponding source system log sequences and target system log sequences; A2. Performing equal partitioning on the source system log sequences and the target system log sequences to obtain log sequence pairs, each log sequence pair including a first subsequence set and a second subsequence set, both of which include multiple log sequences; A3. Performing parsing and transformation processing on the source system log messages and the target system log messages to obtain log template vectors; A4. Performing model training based on the log sequence pairs, the log template vectors, and the total loss function to obtain a trained LSTM model and a hypersphere model; wherein, the total loss function includes: a hypersphere loss function, an alignment loss function, and a uniform loss function, the alignment loss function being used to align or shorten the distance between the same pair of log sequence features, and the uniform loss function being used to make the log sequence features evenly distributed on the hypersphere. The model training method in this application utilizes normal data from the source system and a small amount of normal data from the target system for training, making the target system data more consistent with the data distribution of the source system. This eliminates the need for training with abnormal data from the target system, significantly reducing training costs. Through contrastive learning, it employs alignment loss and uniform loss functions to compare the similarity between two features in pairs. By continuously training the feature extractor, it narrows the differences between features. Directly measuring similarity helps quantify the differences between features, thereby further bridging the gap between the features of the source and target systems. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A schematic diagram illustrating cross-domain model training and log anomaly detection based on transfer learning provided in this application;

[0032] Figure 2 A schematic diagram illustrating the division of log sequences using the sliding window provided in this application;

[0033] Figure 3 A schematic diagram illustrating the data segmentation provided for this application;

[0034] Figure 4 Example diagram of log parsing provided for this application;

[0035] Figure 5 A schematic diagram of the Siamese neural network framework provided in this application;

[0036] Figure 6A schematic diagram of the decision boundary calculation framework provided for this application;

[0037] Figure 7 A flowchart illustrating the steps of the cross-domain model training method based on transfer learning provided in this application;

[0038] Figure 8 A flowchart illustrating the steps of the cross-domain log anomaly detection method based on transfer learning provided in this application;

[0039] Figure 9 A schematic diagram of the terminal device provided in this application.

[0040] Explanation of reference numerals in the attached figures:

[0041] 100. Processor; 200. Memory.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms "a" and "the" as used in the embodiments of this application are also intended to include the plural forms unless the context clearly indicates otherwise.

[0045] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" or "several" means two or more, unless otherwise explicitly specified.

[0046] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which this application can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size should still fall within the scope of the technical content disclosed in this application, provided that they do not affect the effects and purposes that this application can produce.

[0047] It should be understood that the term "and / or" used in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0048] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0049] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0050] Explanation of technical terms:

[0051] LogBERT: A multi-task unsupervised log anomaly detection method based on the BERT architecture. This method uses BERT to randomly initialize and vectorize the parsed log template ID and log template vocabulary to obtain log vectors. Then, by combining two self-supervised training tasks—predicting the log sequence mask ID and minimizing the hypersphere—it learns the pattern of normal log sequences. Log sequences that violate the normal pattern are judged as anomalies. This method is unsupervised.

[0052] WORD2VEC is a group of related models used to generate word vectors. These models are shallow, two-layer neural networks trained to reconstruct linguistic word text. The network represents words and needs to guess the input words in adjacent positions. Under the bag-of-words assumption in Word2Vec, the order of words is unimportant. After training, the Word2Vec model can be used to map each word to a vector, which can be used to represent the relationship between words. This vector is the hidden layer of the neural network.

[0053] DEEPSVDD: It uses neural network training to minimize the sample feature space to divide the hypersphere and determines whether the sample point is abnormal based on the distance.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0055] Figure 1 This diagram illustrates the cross-domain model training and log anomaly detection based on transfer learning provided in this application. Figure 7 The flowchart illustrates the steps of the cross-domain model training method based on transfer learning provided in this application. Figure 8 The flowchart of the cross-domain log anomaly detection method based on transfer learning provided in this application is as follows: Figure 1 , Figure 7 and Figure 8 As shown, this application provides a cross-domain model training and log anomaly detection method based on transfer learning. The offline model training process mainly includes five parts: window partitioning, data partitioning, feature embedding, model training, and decision boundary calculation. The online anomaly detection stage mainly includes three parts: window partitioning, feature embedding, and anomaly detection. The following is a detailed description of each process in log anomaly detection and model training:

[0056] A1. Perform sliding window partitioning on the source system log messages and the target system log messages to obtain the corresponding source system log sequence and target system log sequence;

[0057] In some embodiments, the source system is a long-deployed software system containing a large amount of normal log data S. This application defines the source system log dataset as follows: in Represents a log message, N S Represents D S The total number of log messages in the target system. The target system is a newly deployed system that has only collected a small amount of log data. This application defines the r-th target system log dataset as:

[0058]

[0059] in, It is the total number of log messages for the r-th target system.

[0060] It should be noted that, in this embodiment, the log anomaly detection task detects abnormal log sequences. In some embodiments, window partitioning refers to dividing the log message list into different blocks, which are log sequences, using windows. When an abnormal event occurs, the received log messages are abnormal. If there are abnormal log messages in the log sequence, it indicates that the log sequence is abnormal; if there are no abnormal log messages in the log sequence, it indicates that the log sequence is normal. The log anomaly detection task is to detect log sequences.

[0061] In some embodiments, the sliding window partitioning process, i.e. window partitioning, specifically involves: using a sliding window size W and a step size SP to segment the log messages according to the sequence; a sliding window is based on the idea of ​​two pointers, where a window is formed between the elements pointed to by the two pointers, just like a sliding window nested in a sequence, sliding left and right, and the window contains a set of content; Figure 2 A schematic diagram illustrating the sliding window used to divide the log sequence in this application is shown below. Figure 2 As shown, taking a sliding window size W=4 and a step size SP=1 as an example, the log sequence obtained from the first partition is l1={x1,x2,x3,x4}. Then, the window slides forward according to the step size to obtain the second log sequence l2={x2,x3,x4,x5}. This process continues until the window slides to the last log message.

[0062] Therefore, the i-th log sequence is:

[0063] l i ={x j ,…,x j+W} j＝i*sp

[0064] Where, x j If the j-th log message is represented, then the total number of log sequences after partitioning is:

[0065] M = (NW) / SP + 1

[0066] Where M represents the total number of log sequences, N represents the total number of log messages, W represents the sliding window size, and SP represents the sliding window step size.

[0067] Therefore, the source system log sequence is represented as: The r-th target system log sequence is represented as

[0068] A2. The source system log sequence and the target system log sequence are divided into equal parts to obtain log sequence pairs. The log sequence pairs include a first subsequence set and a second subsequence set. Both the first subsequence set and the second subsequence set include multiple log sequences.

[0069] It should be noted that, in this embodiment, the difference between the source system logs and the target system logs is much greater than the difference between the source system logs themselves. Therefore, to compare the similarity of the log sequence features between the source and target systems, the data needs to be divided into pairs. By comparing these pairs, the overall similarity difference can be obtained. By reducing the contrast loss and adjusting the output of the feature extractor, the difference between the source and target systems is reduced, and the feature similarity of the target system is increased. Therefore, this application mixes the data from the source and target systems and divides them into two sub-log sequence sets: a first sub-sequence set and a second sub-sequence set. These two sub-sequence sets will provide paired log sequences for subsequent downstream tasks.

[0070] To measure the similarity between system data and reduce feature differences, system log data needs to be divided into pairs. In some embodiments, the equal-size partitioning process, i.e., data partitioning, specifically involves: dividing the source system logs and target system logs using a sliding window to obtain the source system log sequence L. S With multiple target system log sequences Figure 3 A schematic diagram of data segmentation provided for this application, such as Figure 3 As shown, the source system log sequence L S With multiple target system log sequences After random shuffling, the datasets are divided into two equal subsets, L1 and L2, which serve as inputs for subsequent models. The log sequences from the source and target systems are not distinguished and are randomly mixed and split.

[0071] Specifically, in this embodiment, the log vector sequence is mixed and then divided into segments of average length. Two subsequence sets.

[0072] The two subsequence sets provide log sequence pairs for the twin LSTM model.

[0073] A3. Parse and transform the source system log messages and the target system log messages to obtain a log template vector;

[0074] It should be noted that logs are unstructured text, requiring the semi-structured log messages to be parsed into structured log templates. The detection model cannot directly process text data; therefore, it needs to extract words from the log templates using WORD2VEC, mapping each word to its corresponding vector representation to construct a log template vector, which serves as input to the detection model.

[0075] Log messages consist of semi-structured constant strings and variables. Before log anomaly detection, log messages need to be correctly parsed into a log template, which is either the constant part or a summary of the log.

[0076] In some embodiments, the parsing process, i.e., log parsing, specifically involves parsing log messages into log templates by deleting parameters. Through log parsing, each log message can be resolved into a unique log event. Log parsing can be implemented using P(x) i ) represents P(x i ) is log message x i The mapping.

[0077] Figure 4 Example diagram of log parsing provided for this application, such as Figure 4 The image shows a BGL log message. The message content is 87 L3 EDRAM ERROR(S)(DCR 0X0157) DETECTED AND CORRECTED OVER 27362SECONDS. The log template is <*><*><*>ERROR(S)(DCR<*>)DETECTED AND CORRECTED OVER<*>SECONDS. Through log parsing, a structured representation of the log message can be obtained, and each log message can be mapped to a unique log event.

[0078] To represent log sequences, we first need to represent words and log events. NLP language models can be used to obtain word vectors or embeddings from the logs, and word embeddings can be further used to construct log event embeddings.

[0079] In some embodiments, the transformation process, i.e., feature representation, specifically involves: using the WORD2VEC language model to capture the relationships between words, and converting a single word W into a D-dimensional embedding Word2Vec(w)∈R. d The log template vector e is obtained by summing and averaging the word vectors. j ∈R d Log template vector e j for:

[0080] e j =F(P(x) j))=Mean(Word2Vec(w)),w∈P(x j )

[0081] Where MEAN represents the summation average, W represents a single word, Word2Vec(w) represents the output of the language model, and P(x) represents the output of the language model. j ) indicates a log template.

[0082] A4. Based on the log sequence pairs, the log template vector, and the total loss function, perform model training to obtain the trained LSTM model and hypersphere model;

[0083] The total loss function includes a hyperspherical loss function, an alignment loss function, and a uniform loss function. The alignment loss function is used to align or bring closer the distance between the same pair of log sequence features, and the uniform loss function is used to make the log sequence features evenly distributed on the hypersphere.

[0084] Siamese neural networks are widely used for tasks involving similarity or comparability between two things, such as signature verification, face verification, image similarity, and sentence similarity. Figure 5 A schematic diagram of the Siamese neural network framework provided in this application is shown below. Figure 5 As shown, a Siamese architecture consists of two neural networks with identical structures and shared weights. These two identical models process similar inputs, extracting feature representation vectors in parallel, making it easier to compare paired samples. Furthermore, the shared weights between the models require fewer training parameters, less training data, and less tendency to overfit, representing a few-shot learning approach. Typically, the model takes two sample pairs as input. Siamese neural networks can be various types of neural networks, such as MLP (Multilayer Perceptron), CNN (Convolutional Neural Network), and RNN (Recurrent Neural Network), depending on the task.

[0085] It should be noted that Siamese neural networks can extract common features from similar samples to the maximum extent. This application uses a Siamese LSTM model to share parameters and extract features from log sequence pairs. Three loss functions are used to constrain the features between log sequence pairs and measure the similarity of paired features. To effectively separate normal and abnormal features, this application employs a hyperspherical loss function, ensuring all normal features are close to the center of the sphere, while abnormal features are discarded. Since the model is trained using a large amount of source system data, it is necessary to align the features of the target system with those of the source system to address the data mismatch. This application uses an alignment loss function to measure the similarity between the two, making the target system data more consistent with the distribution of the source system data, thus achieving model transfer. However, during model transfer, some feature information is inevitably lost. To retain as much original feature information as possible, we introduce a uniform loss function to ensure that paired features are evenly distributed on the hypersphere, preserving the original feature information.

[0086] LSTM is a special type of RNN that controls the transmission state through gating states and has been widely used in time series data. In log anomaly detection, the log sequence consists of a series of log vectors, which are ordered according to their generation time. Therefore, this application uses the LSTM model as the basic model of the Siamese neural network to construct the detection model.

[0087] h j =LSTM(e j ,h j-1 )

[0088] v i =h W

[0089] Among them, e j h is the log template vector of the j-th log template. j-1 It is the (j-1)th LSTM hidden layer vector, and the last hidden vector h w Extract the features of the entire log vector sequence as the log sequence representation v i =h W .

[0090] For two subsets of sequences, the feature of the j-th log sequence from the first subset L1 is represented as: All log sequence features are represented to construct a log sequence feature set V1; the j-th log sequence feature of the second subset L2 is represented as... All log feature sequences represent a log sequence feature set V2.

[0091] The method of minimizing the hypersphere for detecting abnormal log sequences has been widely used in unsupervised log anomaly detection. The DEEPSVDD method uses a trained neural network to minimize the feature space to partition abnormal data. Sample points in the output feature space are mapped onto a hypersphere. Normal data, through training, will be mapped as close as possible to the center C of the hypersphere, while abnormal data will be mapped further away from the center C. By judging the distance of the sample points from the center C, abnormal data is filtered out.

[0092] To make normal data as close as possible to the center of the hypersphere, in some embodiments, the mean squared error of the distance between the sample and the center is used as the loss function:

[0093]

[0094] Wherein, V1 represents the log sequence feature set formed after the first subsequence set is extracted by the LSTM model, and V2 represents the log sequence feature set formed after the second subsequence set is extracted by the LSTM model. i denoted as a single log sequence feature, and C represents the hypersphere center feature.

[0095] Total loss function Loss sum for:

[0096] Loss sum =α1*Loss h +α2*Loss align +Loss uniform

[0097] Where α1 and α2 are the hyperparameters balancing the three loss functions, and Loss align Represents the alignment loss function, Loss uniform Represents the uniform loss function, Loss h This represents the hyperspherical loss function.

[0098] Inspired by contrastive learning, this application introduces alignment loss and uniform loss to assign similar features to similar samples and preserve the feature distribution that contains the most information. Alignment loss aligns or brings the same pair of features closer together, while uniform loss makes the features evenly distributed on the hypersphere.

[0099] Since the training effect of contrastive learning depends on two metrics—alignment and uniformity—alignment means that two opposing samples should be mapped to nearby features, while uniformity means that the feature vectors are roughly evenly distributed on the hypersphere, preserving as much information as possible from the data. Simultaneously optimizing these two metrics during training leads to better performance in downstream tasks. Therefore, this application designs alignment loss functions and uniformity loss functions to continuously optimize these two metrics and construct a good hyperspherical representation.

[0100] In some embodiments, the alignment loss function Loss align for:

[0101]

[0102] in, This represents the i-th log sequence feature in the first log sequence feature set. N represents the i-th log sequence feature in the second log sequence feature set. sub This represents the total number of log sequence features in a single log sequence feature set.

[0103] In some implementations, by using a Gaussian kernel function, i.e., a uniform loss function, defined as the logarithm of the Gaussian kernel function for each pair of samples, the uniformity measure can converge the distribution to a uniform form, providing general applicability even with finite samples. The uniform loss function... uniform for:

[0104]

[0105] Where e represents the base of the natural logarithm. This represents the i-th log sequence feature in the first log sequence feature set. N represents the i-th log sequence feature in the second log sequence feature set. sub This represents the total number of log sequence features in a single log sequence feature set.

[0106] It should be noted that in the embodiments of this application, e is an infinite non-repeating decimal, e≈2.71828.

[0107] A5. Extract the log sequence features based on the LSTM model, adjust and determine the decision boundary, the decision boundary is used to distinguish between normal log sequence features and abnormal log sequence features, and obtain the decision boundary distance, the decision boundary distance is the distance from the decision boundary to the center of the hypersphere model;

[0108] It should be noted that, through training the hypersphere, normal and abnormal features have been effectively separated. The boundary between them needs to be determined to achieve anomaly detection. This is achieved by collecting all log messages from the R-th target system for one day and labeling them as a validation set. This includes pre-labeled normal and abnormal logs. The distance between the features of the normal and abnormal log sequences in the validation set and the center of the hypersphere is calculated to obtain the decision boundary of the r-th target system. The characteristics of the target system log sequence surrounding the hypersphere were successfully divided into two groups: normal and abnormal.

[0109] Specifically, in this embodiment, through model training, the hypersphere effectively clusters normal log sequence features around its center, while abnormal sequence features are discarded by the hypersphere. This application collects log messages from one day of operation of the r-th target system as a validation set. There are normal and abnormal log messages, and the validation set is marked. Figure 6 A schematic diagram of the decision boundary calculation framework provided for this application is shown below. Figure 6 As shown, the log sequence set of the r-th target system is obtained by windowing the validation set. Log sequences The LSTM model, trained using the steps described above, extracts log sequence features. Since the validation set contains anomalous log messages, the anomalous log sequence features extracted from the anomalous log sequences are represented as crosses in the graph, while the normal log sequence features are represented as circles. Normal log sequence features cluster around the center of the hypersphere, while anomalous sequence features are thrown away by the hypersphere. Initially, the average distance between all normal log sequence features and the center of the hypersphere is used as the initial boundary distance. Subsequently, the boundary distance is continuously expanded, with log sequence features smaller than the boundary distance considered normal and those larger than the boundary distance considered anomalous. The AUC (Area Under Curve) of the entire validation set log sequences is calculated. Since the log sequences include both normal and anomalous sequences, the boundary distance that yields the optimal AUC is used as the decision boundary for the r-th target system.

[0110] To elaborate further, the formula for calculating the distance DISTANCE between the log sequence features and the center of the hypersphere is as follows:

[0111]

[0112] Where v represents a single log sequence feature and C represents the center of the hypersphere.

[0113] A6. Input the log template vector into the trained LSTM model to obtain a log sequence feature set, which includes multiple log sequence features; input the log sequence features into the trained hypersphere model to obtain anomaly detection results.

[0114] It should be noted that during the detection phase, entirely new log messages generated by the target system are collected, windowed, and log sequence features are extracted. The distance from the feature vector of the new log sequence to the center of the hypersphere is compared with the decision boundary B to identify abnormal log sequences. The process of each step will be described in detail in the next step.

[0115] During the detection phase, novel log messages generated by the r-th target system are collected, windowed, and their log sequence features are extracted. Similar to calculating the decision boundary, the distance DISTANCE between the log sequence features and the center of the hypersphere, and the distance DISTANCE between the log sequence features and the decision boundary are calculated. Perform a comparison. If the distance to DISTANCE is less than the decision boundary distance... If the log sequence is considered normal, then all log messages in the sequence are considered normal; when the distance to DISTANCE is greater than the decision boundary distance... If the log sequence is abnormal, then the log sequence is considered to be abnormal, meaning that the log sequence is abnormal and there are abnormal log messages in the sequence.

[0116] The specific implementation method is as follows:

[0117] The method of this application is evaluated on three log datasets as follows:

[0118] BGL: An open dataset of 214 days of logs collected from the BLUEGENE / L supercomputer system at Lawrence Livermore National Laboratory (LLNL) in Livermore, California.

[0119] THUNDERBIRD: An open dataset of logs collected from the THUNDERBIRD supercomputer system at Sandia National Laboratories (SNL) in Albuquerque over 244 days, containing 211,212,192 raw log messages. We selected the first 5,000,000 log messages as our dataset, of which 226,753 are anomalous log messages.

[0120] WORDCOUNT is a 3-day log dataset collected by an application running within the Hadoop big data processing framework. It's an application released by Hadoop as an example of MAPREDUCE programming. The WORDCOUNT application analyzes the input file and counts the occurrences of each word. It simulates real-world service failures, injecting deployment failures such as computer shutdown, network disconnection, and full disks.

[0121] Table 1 Overview of the dataset

[0122] DATASETS #OFLOGS #OFANOMALIES BGL 1,200,000 99,677 THUNDERBIRD-MINI 5,000,000 226,753 WORDCOUNT 81,948 4,564

[0123] To verify the rationality and advancement of the proposed method in this application, the anomaly detection method was compared with the machine learning method IM, three transfer learning-based methods LOGTAD, LOGTRANSFER, and DEEPSVDD, and three unsupervised methods DEEPLOG, LOGCLUSTER, and LOGBERT.

[0124] IM: IM extracts workflow invariants from log sequences, revealing the inherent linear characteristics of the program workflow. These invariants can be used to effectively detect anomalies in log sequences.

[0125] DEEPLOG: DEEPLOG uses unsupervised LSTM to capture the sequential patterns of normal log sequences and further identifies abnormal log sequences based on predictions of the following log keys.

[0126] LOGCLUSTER is a clustering-based method that detects anomalous log sequences by comparing them with existing clusters.

[0127] LOGBERT: LOGBERT is a self-supervised framework for log anomaly detection based on Transformer Bidirectional Encoder Representation (BERT). It captures patterns in normal log sequences through two tasks: masked log key prediction and hypersphere volume minimization.

[0128] LOGTAD: LOGTAD uses adversarial domain adaptation to make log data from different systems have a similar distribution, so that the detection model can detect anomalies from multiple systems.

[0129] LOGTRANSFER: LOGTRANSFER can achieve cross-system anomaly detection, but it requires labeled data from both the source and target systems to train a classifier that shares the same fully connected network, rather than the same LSTM network between the source and target systems. It is a semi-supervised transfer learning method.

[0130] DEEPSVDD: DEEPSVDD utilizes neural network training to minimize the sample feature space to divide a hypersphere, and determines whether a sample point is abnormal based on distance. To better reflect the real-world environment of log anomaly detection, this embodiment uses the DEEPSVDD concept to construct a basic LSTM+hypersphere model as a baseline for comparison with the proposed model. Compared to the method proposed in this application, DEEPSVDD lacks the construction of a comparison loss, which will be further discussed in the following experiments.

[0131] The window size for the log sequence was set to 20, and the stride to 3. The word vector dimension for WORD2VEC training was 300, the LSTM hidden layer size was 128, and the learning rates A1 and A2 were set to 10E4 and 10E5, respectively. Normal log data from the source system with a size of 100,000 and normal log data from the target system with a size of 1,000 were selected for training. The impact of the number of LSTM layers and the loss function was investigated experimentally.

[0132] For LOGTRANSFER, the pre-trained model is trained using data from all source systems. When BGL or THUNDERBIRD is the target system, the top 2% of the outliers in the dataset are used as transfer data for LOGTRANSFER; when WORDCOUNT is the target system, the top 5% of the outliers in the dataset are used as transfer data. For DEEPSVDD and LOGTAD, the normal log size of the source system is 100,000, and the normal log size of the target system is 1,000. The single-system unsupervised method divides the training set to the test set in a 4:6 ratio.

[0133] All experiments were run on a 12TH GEN Intel Core i5-12600KF and an NVIDIA GeForce RTX 3070TI.

[0134] To evaluate the model performance of this application, experiments were conducted on cross-system migration effects, cross-domain migration, and multi-objective system migration.

[0135] As shown in Table 2, we compared the cases where BGL and THUNDERBIRD are the source and target systems of each other.

[0136] Table 2 Overall Performance of Cross-System Migration

[0137]

[0138] Based on the above experimental results, as an unsupervised learning method based on transfer learning with few training samples, the proposed method LOGMTC outperforms most single-system unsupervised learning anomaly detection methods in terms of recall and F1 score. When compared with the transfer learning method LOGTRANSFER, even though LOGTRANSFER uses labeled data from the target system for training, the proposed method still outperforms LOGTRANSFER. These experimental results demonstrate that the cross-domain log anomaly detection method based on contrastive learning domain adaptation requires only a smaller training dataset to obtain an ideal anomaly detection model, significantly reducing the model training cost.

[0139] In the case of cross-domain migration, as shown in Table 3, we compared the cases where BGL and THUNDERBIRD are the source systems and WORDCOUNT is the target system.

[0140] Table 3 Overall Performance of Cross-Domain Migration

[0141]

[0142]

[0143] The experimental results clearly show that single-system unsupervised learning methods require a large amount of data for training, and the model needs to find the inherent patterns in the data distribution "from scratch." When training samples are insufficient, model performance will severely degrade. With BGL as the source system and WORDCOUNT as the target system, our proposed method LOGMTC achieves precision, recall, and F1 score of 68.61%, 80.60%, and 74.13%, respectively. It can be seen that LOGTAD, compared to the baseline DEEPSVDD, exhibits a performance decline, showing negative transfer, with a significant drop in precision, recall, and F1 score. This is because, in cross-domain scenarios, adversarial methods cannot effectively obfuscate the feature distributions of the source and target systems, leading to poor hypersphere classification performance. Our method LOGMTC, however, directly calculates the similarity between the source and target systems, bringing their feature distributions closer together and thus avoiding negative transfer.

[0144] Multi-objective system transfer is a further derivation of the traditional single-source system-single-objective system transfer, aiming to train a model that can fully utilize data from a single source domain and multiple target domains, achieving good detection results across different target domains. In our experiments, we did not distinguish between cross-system or cross-domain data as target systems because, in the case of multi-objective systems, we wanted the model to maintain good performance even under more complex and varied conditions. Table 4 shows a comparison of the cases where BGL is the source system, THUNDERBIRD is target system A, and WORDCOUNT is target system B. Table 5 shows a comparison of the cases where THUNDERBIRD BGL is the source system, BGL is target system A, and WORDCOUNT is target system B.

[0145] Table 4 shows the overall migration performance of the multi-objective system using BGL as the source system.

[0146]

[0147]

[0148] Table 5 shows the overall migration performance of the multi-objective system with THUNDERBIRD as the source system.

[0149]

[0150] The experimental results above clearly show that when there are two target systems, DEEPSVDD only guarantees the detection accuracy of one target system, which cannot guarantee the migration effect between the two target systems. The method in this application achieves better results; in the case of multiple target systems, the detection performance of the single target system only decreases slightly. This is because, during the migration of multiple target systems, the method LOGMTC in this application avoids mutual interference between multiple target systems by directly merging the features of the source system and the target system.

[0151] Secondly, this application provides a terminal device. Figure 9 A schematic diagram of the terminal device provided in this application, such as Figure 9 As shown, it includes:

[0152] Memory 200 is used to store computer programs;

[0153] The processor 100 is used to read the computer program in the memory 200 and execute the model training method or the log anomaly detection method.

[0154] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor 100, are used to implement the model training method or the log anomaly detection method.

[0155] It should be further noted that the log anomaly detection method provided in this application can be applied to anomaly detection of logs in multi-target systems. By training data from multiple target systems together, a comprehensive and generalized detection model can be obtained.

[0156] In addition, the method involved in this application can also perform log anomaly detection across domains and multiple systems, and can be used in conjunction with any transfer learning or contrastive learning.

[0157] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0158] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0159] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0160] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A cross-domain model training method based on transfer learning, characterized in that, Applied to log anomaly detection, it includes the following steps: A1. Perform sliding window partitioning on the source system log messages and the target system log messages to obtain the corresponding source system log sequence and target system log sequence; A2. The source system log sequence and the target system log sequence are divided into equal parts to obtain log sequence pairs. The log sequence pairs include a first subsequence set and a second subsequence set. Both the first subsequence set and the second subsequence set include multiple log sequences. A3. Parse and transform the source system log messages and the target system log messages to obtain a log template vector; A4. Based on the log sequence pairs, the log template vector, and the total loss function, perform model training to obtain the trained Long Short-Term Memory (LSTM) network model and the hypersphere model. The total loss function includes: a hyperspherical loss function, an alignment loss function, and a uniform loss function. The alignment loss function is used to align or bring closer the distance between the same pair of log sequence features, and the uniform loss function is used to make the log sequence features evenly distributed on the hyperspherical surface. The process of equal partitioning is as follows: After the source system logs and target system logs are split by sliding window, the source system log sequence and multiple target system log sequences are obtained. The source system log sequence and multiple target system log sequences are mixed and randomly shuffled, and then divided into two equal subsets, which are used as inputs for subsequent models. The log sequences of the source system and the target system are not distinguished and are randomly mixed and split.

2. The cross-domain model training method based on transfer learning according to claim 1, characterized in that, The hyperspherical loss function for: ； in, This represents the first log sequence feature set formed after the first subset of sequences is extracted by the LSTM model. This represents the second log sequence feature set formed after the second subset of sequences is extracted by the LSTM model. This represents the feature of a single log sequence, where C represents the hyperspherical center feature. The total loss function for: ； in, , These are the hyperparameters that balance the three loss functions. Represents the alignment loss function. Represents the uniform loss function. This represents the hyperspherical loss function.

3. The cross-domain model training method based on transfer learning according to claim 2, characterized in that, The alignment loss function for: ； in, This represents the i-th log sequence feature in the first log sequence feature set. This represents the i-th log sequence feature in the second log sequence feature set. This represents the total number of log sequence features in a single log sequence feature set.

4. The cross-domain model training method based on transfer learning according to claim 2 or 3, characterized in that, The uniform loss function for: ； in, The base of the natural logarithm. This represents the i-th log sequence feature in the first log sequence feature set. This represents the i-th log sequence feature in the second log sequence feature set. This represents the total number of log sequence features in a single log sequence feature set.

5. The cross-domain model training method based on transfer learning according to claim 1, characterized in that, The A2 includes: A21. Mix the source system log sequence and the target system log sequence and randomly shuffle them to obtain a mixed log sequence; A22. Divide the mixed log sequence into two equal sets of sub-log sequences to obtain the log sequence pair.

6. The cross-domain model training method based on transfer learning according to claim 1, characterized in that, Also includes: Based on the LSTM model, the log sequence features are extracted, and the decision boundary is adjusted and determined. The decision boundary is used to distinguish between normal log sequence features and abnormal log sequence features. The decision boundary distance is obtained, which is the distance from the decision boundary to the center of the hypersphere model.

7. A cross-domain log anomaly detection method based on transfer learning, characterized in that, The log anomaly detection method is implemented using an LSTM model and a hypersphere model, wherein the LSTM model and the hypersphere model are trained using the model training method according to any one of claims 1-6, and the log anomaly detection method includes the following steps: B1. Perform sliding window segmentation on the log messages of the target system to be detected to obtain the target system log sequence; B2. Parse the target system log sequence into a log template, and obtain a log template vector based on the log template; B3. Input the log template vector into the trained LSTM model to obtain a log sequence feature set, which includes multiple log sequence features. B4. Input the log sequence features into the trained hypersphere model to obtain the anomaly detection results.

8. The cross-domain log anomaly detection method based on transfer learning according to claim 7, characterized in that, The B4 includes: B41. Calculate the first distance from the log sequence features to the center of the hyperspherical model; B42. Compare the first distance with the decision boundary distance to obtain the comparison result; B43. Based on the comparison results, the anomaly detection results are obtained.

9. A terminal device, characterized in that, include: Memory, used to store computer programs; A processor is configured to read a computer program from the memory and execute the cross-domain model training method based on transfer learning as described in any one of claims 1-6 or the cross-domain log anomaly detection method based on transfer learning as described in any one of claims 7-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the cross-domain model training method based on transfer learning as described in any one of claims 1-6 or the cross-domain log anomaly detection method based on transfer learning as described in any one of claims 7-8.

Images