Network log anomaly detection method and system based on GPT2-PCA

Abstract

The application discloses a network log anomaly detection method and system based on GPT2-PCA, collects log messages; the log messages are preprocessed to obtain a log message set; the log messages are structured to obtain a structured log set; serialization is performed to obtain a serialized log set; a GPT-2 model is pre-trained through the serialized log set, a trained GPT-2 model is used for feature extraction of the log messages, and a log embedding vector set is obtained; a classification model based on PCA is trained according to the log embedding vector set, and the trained classification model based on PCA is used for anomaly detection. The application can support fine-grained preprocessing of logs of various devices and applications, can capture normal behavior patterns of logs in a longer context, and can effectively improve the speed and accuracy of detection.

Description

Technical Field

[0001] This invention belongs to the field of log anomaly detection, specifically relating to a network security log anomaly detection method and system based on GPT2-PCA. Background Technology

[0002] Advanced Persistent Threats (APTs) are sophisticated cyberattack patterns executed by highly skilled attackers, aiming to remain entrenched within a target network for extended periods, conducting continuous intelligence gathering or data theft. Investigating APT attacks requires analyzing countless logs. Enterprise intrusion detection systems generate numerous threat intelligence alerts for suspicious events. Security personnel need to analyze these alerts to determine if they are part of an attack, but due to the sheer volume of logs, they are prone to alert fatigue, neglecting many alerts and missing genuine attack events.

[0003] Existing methods primarily utilize unsupervised learning to train deep learning models on benign log files. These methods merge logs from various devices and applications, generate timestamps for log messages based on date and time, sort them by timestamp, and use regular expressions to process numeric variables in the log messages, forming a structured log sequence. A BERT model is pre-trained using benign logs for feature extraction, resulting in embedding vectors. Anomaly detection is then performed using deep learning or machine learning models.

[0004] However, existing technologies have the following limitations:

[0005] 1. The preprocessing granularity of logs is too large. Sorting different types of logs by timestamp may lead to false alarms and missed alarms. Different logs are recorded from different devices and record different events. Simply sorting them by timestamp may introduce a lot of noise into the logs and destroy the original log structure.

[0006] 2. Use the BERT model to extract features from logs. This method mainly focuses on whether a single log message itself is abnormal. BERT has a weak ability to capture anomalies at the sequence level and cannot detect anomalies at the log sequence level.

[0007] 3. Use deep learning and machine learning models for anomaly detection. Deep learning models are too slow when dealing with logs in production environments; ordinary machine learning models, in unsupervised learning scenarios, mainly rely on model convergence to determine the decision boundary, but unclean datasets can lead to inaccurate decision boundaries, causing a decline in the model's classification performance. Summary of the Invention

[0008] Purpose of the invention: To address the problems of excessively large granularity in log preprocessing, inability to detect anomalies at the log sequence level, slow processing speed, and degraded classification performance, this invention provides a network log anomaly detection method based on GPT2-PCA. This method can perform fine-grained analysis on logs from APT attack investigations, capture normal behavior patterns in logs within a longer context, and improve the accuracy and speed of anomaly detection. Furthermore, the solution has a simple structure and is easy to implement.

[0009] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0010] A network log anomaly detection method based on GPT2-PCA includes the following steps:

[0011] Step S1: Collect log messages. Preprocess the log messages to obtain a log message set.

[0012] Step S2: Represent the log messages in a structured manner according to the log message set to obtain a structured log set.

[0013] Step S3: Serialize the log messages according to the structured log set to obtain a serialized log set.

[0014] Step S4: Pre-train the GPT-2 model using the serialized log set, and use the trained GPT-2 model to extract features from the log messages to obtain a set of log embedding vectors.

[0015] Step S5: Train a PCA-based classification model based on the log embedding vector set, and use the trained PCA-based classification model for anomaly detection.

[0016] Preferred method: In step S4, the GPT-2 model is pre-trained using a serialized log set, and the trained GPT-2 model is used to extract features from log messages. This involves training the GPT-2 model with a large number of unlabeled logs, and then using the trained GPT-2 model to extract features from the log sequence to be detected.

[0017] The objective function of the pre-trained GPT-2 model is defined as follows:

[0018]

[0019] in, This represents the objective loss function, where N is the number of log sequences and T is the length of each sequence. Indicates in a given sequence In the case of GPT-2 predicting the log key at position t+1, This indicates that the (t+1)th position is the log key k. i , Represents the 1 to t log keys of a given sequence i.

[0020] After pre-training, the GPT-2 model can be based on a given log sequence fragment. Generate a log sequence This indicates that the log key at the predicted T-th position is k. i T represents the length of the sequence.

[0021] Input the serialized log S to be detected into the pre-trained GPT-2 model, obtain the last hidden layer of the GPT-2 model output, and then extract each sequence S. i The vector v of the last token i As semantic vectors, they form a log embedding vector set V, V = {v1, v2, ..., v...} N}, where N represents the number of log sequences.

[0022] The preferred structure of the GPT-2 model is as follows:

[0023]

[0024] in, This indicates the use of the Transformer decoder. Decode it. This represents the hidden layer representation derived from the Transformer decoder. Indicates the hidden state The scores generated after transformation by the weight matrix W are normalized using the softmax function to obtain a probability distribution. Indicates in a given sequence In the case of , the probability of the log key predicted by GPT-2 at position t+1. These are the header parameters of the language model. Let d represent the real number field and d represent the length of the vector. This represents a set containing all possible log keys.

[0025] Preferred method: The method for anomaly detection using the trained PCA-based classification model in step S5 is to construct a normal sample space S. n and an anomaly sample space S a Then each log sequence vector v i Through formula Projected onto normal space S n Above, where P = [e1, e2, ..., e k ],e1,e2,…e kThis represents the feature vector with the top k largest eigenvalues, where k is the number of principal components in the log sequence. For v i In normal space S n The projection onto v. Similarly, v i Through formula Projected onto the anomaly space S a Above, where I is the standard identity matrix. The squared prediction error is used to determine the incoming log sequence v. j Is it an outlier in the anomalous space? j The SPE value is obtained through the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j It was predicted to be an abnormal log sequence.

[0026] Preferred method: The method for preprocessing log messages in step S1 is to parse the original log content according to the log format, extract semi-structured log messages, and form a log message set L, L = {l1, l2, ..., l...} n}, where n represents the size of the log message set, l n This represents the nth semi-structured log message.

[0027] Preferably, the specific method for structuring the log messages in step S2 includes the following steps:

[0028] Step S2.1: Downsample the log messages by a certain percentage using the hyperparameter r to form sampled log L. sample .

[0029] Step S2.2, sample log L sample Use the Drain3 log parser to parse the logs and extract the log template set K. sample And save the set L of unmatched logs. unmatch .

[0030] Step S2.3, again check the unmatched log set L unmatch The newly extracted template is obtained by template parsing, and then added to K. sample Form a new log template set K, K = {k1,k2,…,k m}, where m represents the size of the log template set, and k m This represents the m-th log template.

[0031] Step S2.4: Use the log template set K to perform template matching on the full log set L to form a structured log set D, D = {d1, d2, ..., d...} n}, where n represents the size of the structured log collection, and dn This represents the nth structured log entry.

[0032] Preferably, the specific method for serializing log messages in step S3 is to use a sliding time window method, grouping the structured log sequence into event sequences by using the window size w and the step size s. The structured log set D of length n, D = {d1, d2, ..., dn} n}, then the i-th window can be represented as:

[0033] s i ={d is+1 d is+2 ,…,d (is+w)}

[0034] Where i represents the window index, starting from 0.

[0035] Finally, a serialized log set S is formed, S = {s1, s2, ..., s...} N},

[0036] Preferably, the log messages include operating system logs, application logs, and other logs. Operating system logs refer to logs generated by the operating system itself. Application logs refer to logs generated by applications or other related programs running on the operating system. Other logs refer to logs that do not contain abnormal information but contain other important information.

[0037] Another objective of this invention is to provide a network log anomaly detection system based on GPT2-PCA, used to implement the aforementioned network log anomaly detection method based on GPT2-PCA, comprising a data acquisition unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein:

[0038] The collection unit is used to collect log messages.

[0039] The preprocessing unit is used to preprocess the log messages to obtain a log message set.

[0040] The structured log unit is used to represent log messages in a structured manner according to the log message set to obtain a structured log set.

[0041] The serialization log unit is used to serialize log messages based on the structured log set to obtain a serialized log set.

[0042] The GPT-2 model unit is used to pre-train a GPT-2 model using a serialized log set, and then use the trained GPT-2 model to extract features from the log messages to obtain a log embedding vector set.

[0043] The PCA-based classification model unit is used to train a PCA-based classification model based on the log embedding vector set, and to use the trained PCA-based classification model for anomaly detection.

[0044] The output unit is used to output anomaly detection.

[0045] Another object of the present invention is to provide a computer system, including a memory and a processor, wherein the memory is used to store computer programs / instructions. The processor is used to execute the computer programs / instructions to implement the GPT2-PCA-based network log anomaly detection method described above.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] (1) The present invention can perform fine-grained processing on various logs from different devices and applications, and improve the processing speed, which is beneficial to subsequent training and detection.

[0048] (2) Using GPT-2 to extract the embedding vector of log sequences can capture the behavioral patterns of log sequences in a longer context.

[0049] (3) Using the PCA classification model to classify the log vectors to be detected results in faster detection speed, higher accuracy, and ease of implementation and application. Attached Figure Description

[0050] Figure 1 This is the overall flowchart of the present invention.

[0051] Figure 2 This is a schematic diagram of the Firefox log preprocessing of the present invention.

[0052] Figure 3 This is a schematic diagram of the Windows security log preprocessing of the present invention.

[0053] Figure 4 This is a schematic diagram of DNS log preprocessing according to the present invention. Detailed Implementation

[0054] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0055] Example 1

[0056] A network log anomaly detection method based on GPT2-PCA, such as Figure 1 As shown, it includes the following steps:

[0057] Step S1: Collect log messages. Preprocess the log messages to obtain a log message set.

[0058] The log messages include operating system logs, application logs, and other logs. Operating system logs refer to logs generated by the operating system itself. Application logs refer to logs generated by applications or other related programs running on the operating system. Other logs refer to logs that do not contain exception information but contain other important information.

[0059] The method for preprocessing log messages in step S1 is to parse the original log content according to the log format such as: <date><time><level><source><content>, extract semi-structured log messages, and form a log message set L, L={l1,l2,…,l n}, where n represents the size of the log message set, l n This represents the nth semi-structured log message.

[0060] Step S2: Represent the log messages in a structured manner according to the log message set to obtain a structured log set.

[0061] The specific method for representing log messages in a structured manner includes the following steps:

[0062] Step S2.1: Downsample the log messages by a certain percentage using the hyperparameter r to form sampled log L. sample .

[0063] Step S2.2, sample log L sample Use the Drain3 log parser to parse the logs and extract the log template set K. sample And save the set L of unmatched logs. unmatch .

[0064] Step S2.3, again check the unmatched log set L unmatch The newly extracted template is obtained by template parsing, and then added to K. sample Form a new log template set K, K = {k1,k2,…,k m}, where m represents the size of the log template set, and k m This represents the m-th log template.

[0065] Step S2.4: Use the log template set K to perform template matching on the full log set L to form a structured log set D, D = {d1, d2, ..., d...} n}, where n represents the size of the structured log collection, and d n This represents the nth structured log entry.

[0066] Step S3: Serialize the log messages according to the structured log set to obtain a serialized log set.

[0067] The specific method for serializing log messages is to use a sliding time window approach, grouping the structured log sequence into event sequences based on the window size w and step size s. We have a structured log set D of length n, D = {d1, d2, ..., dn}. n}, then the i-th window can be represented as:

[0068] s i ={d is+1 ,d is+2 ,…,d (is+w)}

[0069] Where i represents the window index, starting from 0.

[0070] Finally, a serialized log set S is formed, S = {s1, s2, ..., s...} N},

[0071] Step S4: Pre-train the GPT-2 model using the serialized log set, and use the trained GPT-2 model to extract features from the log messages to obtain a set of log embedding vectors.

[0072] The method of pre-training a GPT-2 model using a serialized log set and then using the trained GPT-2 model to extract features from log messages involves training the GPT-2 model with a large number of unlabeled logs to enable the GPT-2 model to capture the normal behavior patterns of log sequences, and then using the trained GPT-2 model to extract features from the log sequences to be detected.

[0073] GPT-2, based on the Transformer decoder, utilizes a self-attention mechanism to capture the dependencies between log keys in a log sequence. GPT-2 is trained to predict the next log key given a previous log key. The objective function of the pre-trained GPT-2 model is defined as follows:

[0074]

[0075] in, This represents the objective loss function, where N is the number of log sequences and T is the length of each sequence. Indicates in a given sequence In the case of GPT-2 predicting the log key at position t+1, This indicates that the (t+1)th position is the log key k. i , Represents the 1 to t log keys of a given sequence i.

[0076] In order to derive The structure of the GPT-2 model is as follows:

[0077]

[0078] in, This indicates the use of the Transformer decoder. Decode it. This represents the hidden layer representation derived from the Transformer decoder. Indicates the hidden state The scores generated after transformation by the weight matrix W are normalized using the softmax function to obtain a probability distribution. Indicates in a given sequence In the case of , the probability of the log key predicted by GPT-2 at position t+1. These are the header parameters of the language model. Let d represent the real number field and d represent the length of the vector. This represents a set containing all possible log keys.

[0079] After pre-training, the GPT-2 model can be based on a given log sequence fragment. Generate a log sequence This means the model has learned the pattern of normal logs. This indicates that the log key at the predicted T-th position is k. i T represents the length of the sequence.

[0080] Input the serialized log S to be detected into the pre-trained GPT-2 model, obtain the last hidden layer of the GPT-2 model output, and then extract each sequence S. i The vector v of the last token i As a semantic vector, this is because the last token contains all the semantics of the entire sequence, forming a log embedding vector set V, V = {v1, v2, ..., v...} N}, where N represents the number of log sequences.

[0081] Step S5: Train a PCA-based classification model based on the log embedding vector set, and use the trained PCA-based classification model for anomaly detection. Combine the anomaly detection results from all logs to obtain the final attack investigation results.

[0082] The method for training a PCA-based classification model based on a set of log embedding vectors, and then using the trained PCA-based classification model for anomaly detection, involves constructing a normal sample space S. n and an anomaly sample space Sa Then each log sequence vector v i Through formula Projected onto normal space S n Above, where P = [e1, e2, ..., e k ],e1,e2,…e k This represents the eigenvector with the top k largest eigenvalues, where k is the number of principal components in the log sequence. This also indicates that the log sequence has k principal components. For v i In normal space S n The projection onto the surface. Similarly, v will also be... i Through formula Projected onto the anomaly space S a Above, where I is the standard identity matrix. Since the components corresponding to the outlier space retain the minimum variance of the original data, outliers can be detected by detecting them. The squared prediction error is used to determine the incoming log sequence v. j Is it an outlier in the anomalous space? j The SPE value is obtained through the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j If a log sequence is predicted as an anomalous sequence, SPE (Square PredictionError) represents the squared prediction error.

[0083] This method supports fine-grained preprocessing of logs from various devices and applications, captures normal behavior patterns in logs within a longer context, and effectively improves detection speed and accuracy.

[0084] Example 2

[0085] This embodiment provides a network log anomaly detection system based on GPT2-PCA, used to implement the network log anomaly detection method based on GPT2-PCA. It includes a data acquisition unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein:

[0086] The collection unit is used to collect log messages.

[0087] The preprocessing unit is used to preprocess the log messages to obtain a log message set.

[0088] The structured log unit is used to represent log messages in a structured manner according to the log message set to obtain a structured log set.

[0089] The serialization log unit is used to serialize log messages based on the structured log set to obtain a serialized log set.

[0090] The GPT-2 model unit is used to pre-train a GPT-2 model using a serialized log set, and then use the trained GPT-2 model to extract features from the log messages to obtain a log embedding vector set.

[0091] The PCA-based classification model unit is used to train a PCA-based classification model based on the log embedding vector set, and to use the trained PCA-based classification model for anomaly detection.

[0092] The output unit is used to output anomaly detection.

[0093] Example 3

[0094] This embodiment provides a computer system, including a memory and a processor. The memory stores computer programs / instructions. The processor executes the computer programs / instructions to implement the GPT2-PCA-based network log anomaly detection method.

[0095] Example 4

[0096] This embodiment provides a specific example of a network log anomaly detection method based on GPT2-PCA, such as... Figure 1 As shown, consider a multi-device, multi-application log anomaly detection scenario in APT attack incident response or log auditing. Based on their source and function, logs can be categorized into three types: operating system logs, application logs, and other logs. Operating system logs refer to logs generated by the operating system itself; application logs refer to logs generated by applications or other related programs running on the operating system; and other logs refer to logs that do not contain anomaly information but contain other important information.

[0097] The overall anomaly detection consists of three steps:

[0098] Step 1: Process the logs, which includes three steps: log parsing, template matching, and log grouping.

[0099] Step 2: Train the anomaly detection model. The log sequences are segmented. Then, a large number of unlabeled samples are used to train the GPT-2 model to capture normal patterns in the logs. Finally, a PCA classifier, an unsupervised machine learning model, is trained using the dataset.

[0100] Step 3: Log Anomaly Detection. For the logs to be detected, the logs are first preprocessed, then features are extracted using a trained GPT-2 model, and finally, a PCA classifier is used for classification to identify abnormal logs. The detection results of different logs are then combined for attack investigation.

[0101] In this embodiment of the invention, the original log dataset comes from the ATLAS dataset, which contains logs from attacked hosts. Each host has three types of logs collected: Windows security logs, Firefox browser logs, and DNS resolution logs. These belong to operating system logs, application logs, and other logs, respectively.

[0102] Step 1: Log Preprocessing. This dataset provides logs from three different applications. The log preprocessing method can be adjusted according to the different log formats. Figure 2 , 3 Figures 4 and 5 respectively demonstrate the preprocessing methods and results for three different logs.

[0103] Figure 2 This demonstrates a method for preprocessing Firefox logs. The raw log content is processed according to the log format. <date> <time> <utc> -[ <thread> ]: <level> / <component> <content>The original log is parsed to extract semi-structured log messages, which are the messages in the original log. <content>Partially, forming a log message set L, L = {l1, l2, ..., l...} n Log messages L are controlled via hyperparameter r. Firefox Perform downsampling at a certain ratio to form L sample For sampling log L sample Use the Drain3 log parser to parse the logs and extract the log template set K. sample And save the set L of unmatched logs. unmatch Again, for the unmatched log set L unmatch Perform template parsing and add the newly extracted template to K. sample This forms a new log template set K, K = {k1, k2, ..., k m The log template set K is used to perform template matching on the full log set L to form a structured log set D, D = {d1, d2, ..., dn}. n Using a sliding time window technique, the structured log sequence is grouped into event sequences by the window size w and the step size s. The i-th window can be represented as: s i ={d is+1 ,d is+2 ,…,d (is+w) }, where i represents the window index, starting from 0. Finally, a serialized log set S is formed. Firefox S firefox ={s1,s2,…,s N },

[0104] Figure 3 This demonstrates a preprocessing method for Windows security logs. For Windows security log messages L... WinSec ={l1,l2,…,l n Extract the event ID from each log message. This is a Windows security log identifier used to identify specific event types. Each ID represents a different log event, forming a structured log set D, where D = {d1, d2, ..., dn}. n Logs generated by the same process are grouped together to form a serialized log set S, S = {s1, s2, ..., s3}. k }, where k is the number of different processes in the log message set, s i ={d1,d2,…,d m }, where m is the number of logs generated by the process. A sliding window technique is used to prune excessively long log sequences. Within a specified time window size w, log sequences s longer than w are pruned. i Cut to Each sub-window. Finally, a serialized log set S is formed. WinSec S WinSec ={d1,d2,…,d N }

[0105] Figure 4 This demonstrates a method for preprocessing DNS resolution logs. A dictionary is used to store the mapping between domain names and IP addresses in the DNS logs, and the data is then converted to JSON format and stored in a file.

[0106] Step 2: Train the anomaly detection model. Train the GPT-2 model using a large number of unlabeled logs to enable GPT-2 to capture the behavioral patterns of benign logs. Then, use the GPT-2 model to extract features from the log sequences to be detected.

[0107] GPT-2, based on the Transformer decoder, utilizes a self-attention mechanism to capture the dependencies between log keys in a log sequence. GPT-2 is trained to predict the next log key given a previous log key. The objective function for pre-trained GPT-2 is defined as follows:

[0108]

[0109] Where N is the number of log sequences, and T is the length of each sequence. Indicates in a given sequence In the case of , the probability of the log key predicted by GPT-2 at position t+1.

[0110] In order to derive The structure of GPT2 is as follows:

[0111]

[0112]

[0113] in This represents the hidden representation obtained from the Transformer decoder, while It is a header parameter of the language model, which maps the hidden representation to all log keys. The probability distribution.

[0114] After pre-training, GPT-2 can be based on given log sequence fragments. Generate a log sequence This means the model has learned the pattern of normal logs.

[0115] In this example, GPT-2 is trained using unlabeled Firefox logs and Windows security logs. After training, the trained GPT-2 model is used to extract features from these benign logs. The serialized logs S are input into the model, the last hidden layer of the model's output is obtained, and then each sequence S is extracted. i The vector v of the last token i This serves as a semantic vector because the last token contains the complete semantics of the entire sequence. This forms a log embedding vector set V, where V = {v1, v2, ..., v...}. N }

[0116] Then, a PCA-based classification model is trained using the log embedding vector set V. A normal sample space (using S) is constructed. n (represented by S), and an anomaly sample space (represented by S). a (represented), then each log sequence vector (using v) i (represented by formula) Projected into normal space Above. Where P = [e1, e2, ..., e k ], and e1, e2, ... e k This represents the eigenvector with the largest eigenvalue. This also indicates that the log sequence has k principal components. For v i In normal space S n The projection onto the surface. Similarly, v will also be... i Through formula Projected onto the anomaly space S a Let I be the standard identity matrix. Since the components corresponding to the outlier space retain the minimum variance of the original data, outliers can be detected by detecting outliers in the outlier space.

[0117] Step 3: Log Anomaly Detection. The logs to be detected are preprocessed using Step 1 to form the log sequence S′ to be detected. The pre-trained GPT-2 model is then used to extract features from the log sequence in Step 2, forming the embedding vector V′. The PCA-based classification model trained in Step 2 is then used for classification. The squared prediction error is used to determine the input log sequence (using v′). j This indicates whether a point is an outlier in the anomaly space. j The SPE value is obtained through the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j It was predicted to be an abnormal log sequence.

[0118] The obtained abnormal log sequences are aggregated and mapped to structured log messages D via log IDs. Entities such as URLs and processes are extracted from the malicious structured log parameter list to obtain malicious entities. The relationships between entities are uncovered through the mapping in DNS logs to conduct a comprehensive attack investigation while eliminating false positives.

[0119] This invention proposes a fine-grained preprocessing method for various logs from different devices and applications, improving processing speed and addressing the problem of excessively coarse granularity in existing log preprocessing methods, which is beneficial for subsequent training and detection. It also proposes an extraction method using GPT-2 to extract log sequence embedding vectors, which can capture behavioral patterns in log sequences within a longer context, addressing the weakness of existing feature extraction methods in capturing anomalies in sequences. Finally, it proposes a method using a PCA classification model to classify the log vectors to be detected, achieving fast classification speed and higher accuracy, and being easy to implement and apply, solving the problems of slow classification speed and low classification performance in existing classification models.

[0120] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / content> < / content> < / component> < / level> < / thread> < / utc> < / time> < / date>

Claims

1. A network log anomaly detection method based on GPT2-PCA, characterized in that, Includes the following steps: Step S1: Collect log messages; preprocess the log messages to obtain a log message set; Step S2: Represent the log messages in a structured manner according to the log message set to obtain a structured log set; Step S3: Serialize the log messages according to the structured log set to obtain a serialized log set; Step S4: Pre-train the GPT-2 model using the serialized log set, and use the trained GPT-2 model to extract features from the log messages to obtain a set of log embedding vectors; The method of pre-training a GPT-2 model using a serialized log set and then using the trained GPT-2 model to extract features from log messages is to train the GPT-2 model with a large number of unlabeled logs and then use the trained GPT-2 model to extract features from the log sequence to be detected. The objective function of the pre-trained GPT-2 model is defined as follows: in, Represents the target loss function. It is the number of log sequences. It is the length of each sequence. Indicates in a given sequence In the case of GPT-2 in the 19th century The probability of predicting the log key at each location. This indicates that the (t+1)th position is the log key. , Represent a given sequence of to One log key; After pre-training, the GPT-2 model can be based on a given log sequence fragment. Generate a log sequence , Indicates the predicted first The log key for each position is , Indicates the length of the sequence; Serialized logs to be detected Input the pre-trained GPT-2 model, obtain the last hidden layer of the GPT-2 model output, and then extract each sequence. The vector of the last token As semantic vectors, they form a log embedding vector set. , , Indicates the number of log sequences; The structure of the GPT-2 model is as follows: in, This indicates the use of the Transformer decoder. Decode it. This represents the hidden layer representation derived from the Transformer decoder. Indicates the hidden state After weight matrix The normalization function is applied to the generated fractions after the transformation. This yields a probability distribution. Indicates in a given sequence In the case of GPT-2 in the 19th century The probability of predicting the log key at each location. These are the header parameters of the language model. Represents the real number field. Represents the length of the vector. This represents a set containing all possible log keys; Step S5: Train a PCA-based classification model based on the log embedding vector set, and use the trained PCA-based classification model for anomaly detection. The method for anomaly detection using a pre-trained PCA-based classification model is to construct a normal sample space. and an abnormal sample space Then each log sequence vector Through formula Projected into normal space Above, among which, , Indicates having a previous Eigenvectors with large eigenvalues The number of principal components in the log sequence. for In normal space The projection on; similarly, will Through formula Projected into the anomaly space Above, among which The standard identity matrix is ​​used; the squared prediction error is used to determine the incoming log sequence. Is it an outlier in the abnormal space? The SPE value is obtained through the formula To calculate, by comparing with a given threshold Compare, if the SPE value is greater than ,but It was predicted to be an abnormal log sequence.

2. The network log anomaly detection method based on GPT2-PCA according to claim 1, characterized in that: The method for preprocessing log messages in step S1 is to parse the original log content according to the log format, extract semi-structured log messages, and form a log message set. , , Indicates the size of the log message set. Indicates the first A semi-structured log message.

3. The network log anomaly detection method based on GPT2-PCA according to claim 2, characterized in that: The specific method for structuring the log messages in step S2 includes the following steps: Step S2.1, through hyperparameters A certain percentage of log messages are downsampled to form sampled logs. ; Step S2.2, sample log Use the Drain3 log parser to parse logs and extract a set of log templates. And save the collection of unmatched logs. ; Step S2.3, again check the unmatched log set Perform template parsing to obtain the newly extracted template, and add the newly extracted template to... Create a new set of log templates , , Indicates the size of the log template collection. Indicates the first One log template; Step S2.4, use the log template set For the full log collection Perform template matching to form a structured log collection. , , Indicates the size of the structured log collection. Indicates the first Structured log entries.

4. The network log anomaly detection method based on GPT2-PCA according to claim 3, characterized in that: The specific method for serializing log messages in step S3 is to use a sliding time window method, adjusting the window size accordingly. and step length Group the structured log sequence into event sequences; length is Structured log collection , Then the first A window can be represented as: in, Indicates the window index. Start from 0; Finally, a serialized log collection is formed. , , .

5. The network log anomaly detection method based on GPT2-PCA according to claim 4, characterized in that: The log messages include operating system logs, application logs, and other logs. The operating system logs refer to logs generated by the operating system itself; the application logs refer to logs generated by applications or other related programs running on the operating system; and the other logs refer to logs that do not contain abnormal information but contain other important information.

6. A network log anomaly detection system based on GPT2-PCA, characterized in that, The network log anomaly detection method based on GPT2-PCA as described in any one of claims 1-5 includes a collection unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein: The acquisition unit is used to collect log messages; The preprocessing unit is used to preprocess the log messages to obtain a log message set; The structured log unit is used to represent log messages in a structured manner according to the log message set to obtain a structured log set; The serialization log unit is used to serialize log messages according to the structured log set to obtain a serialized log set; The GPT-2 model unit is used to pre-train a GPT-2 model using a serialized log set, and then use the trained GPT-2 model to extract features from the log messages to obtain a log embedding vector set. The PCA-based classification model unit is used to train a PCA-based classification model based on the log embedding vector set, and to use the trained PCA-based classification model for anomaly detection. The output unit is used to output anomaly detection.

7. A computer system, characterized in that, It includes a memory and a processor, wherein the memory is used to store computer programs / instructions; and the processor is used to execute the computer programs / instructions to implement the network log anomaly detection method based on GPT2-PCA as described in any one of claims 1-5.

Images