Weblog anomaly detection method and system based on GPT2-PCA

Abstract

The invention discloses a network log anomaly detection method and system based on GPT2-PCA. The method comprises the following steps: collecting a log message; preprocessing the log messages to obtain a log message set; performing structured representation on the log message to obtain a structured log set; serializing to obtain a serialized log set; pre-training a GPT-2 model through the serialized log set, and performing feature extraction on log messages by using the trained GPT-2 model to obtain a log embedding vector set; and training a PCA-based classification model according to the log embedding vector set, and carrying out anomaly detection by using the trained PCA-based classification model. According to the method, fine-grained preprocessing of logs of various devices and applications can be supported, the normal behavior mode of the logs can be captured in a longer context, and the detection speed and precision can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of log anomaly detection, and specifically relates to a network security log anomaly detection method and system based on GPT2-PCA. Background Art

[0002] Advanced persistent threats (APTs) are a complex network attack pattern carried out by highly technical attackers, aiming to lurk inside the target network for a long time and conduct continuous intelligence collection or data theft. In order to investigate APT attacks, countless logs need to be analyzed. The intrusion detection systems deployed by enterprises generate a lot of threat intelligence about suspicious events. Security personnel need to analyze whether these alerts are part of an attack, but due to the large number of logs, security personnel are prone to alarm fatigue, ignoring a large number of alerts and missing real attack events.

[0003] The existing methods mainly use unsupervised learning to train deep learning models on benign log files. The existing methods merge the collected logs of multiple devices and applications, generate timestamps for log messages based on date and time, sort them by timestamps, and use regular expressions to process variables such as numbers in log messages to form structured log sequences. Use benign logs to pre-train the BERT model for feature extraction and form an embedding vector. Use deep learning models or machine learning models for anomaly detection.

[0004] However, existing technologies have the following limitations:

[0005] 1. The granularity of log preprocessing is too large. Sorting different types of logs by timestamp may lead to false positives and false negatives. Different logs are recorded from different devices and events. Simply sorting by timestamp may introduce a lot of noise into the logs and destroy the original log structure.

[0006] 2. Use the BERT model to extract features from logs. This method mainly focuses on whether a single log message itself is abnormal. BERT has a weak ability to capture anomalies in sequences and cannot detect anomalies at the log sequence level.

[0007] 3. Use deep learning and machine learning models for anomaly detection. Deep learning models process logs in production environments too slowly; ordinary machine learning models rely mainly on model convergence to determine decision boundaries in unsupervised learning scenarios, but unclean data sets can lead to inaccurate decision boundaries and reduce the model's classification performance. Summary of the invention

[0008] Purpose of the invention: In order to solve the problems of too large granularity of log preprocessing, inability to detect anomalies at the log sequence level, slow processing speed and reduced classification performance, the present invention provides a network log anomaly detection method based on GPT2-PCA, which can perform fine-grained analysis on logs of APT attack investigations, capture normal behavior patterns of logs in a longer context, and improve the accuracy and speed of anomaly detection. The scheme has a simple structure and is easy to implement.

[0009] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0010] A network log anomaly detection method based on GPT2-PCA, comprising the following steps:

[0011] Step S1, collecting log messages. Preprocessing the log messages to obtain a log message set.

[0012] Step S2: According to the log message set, the log message is structured to obtain a structured log set.

[0013] Step S3: Serialize the log message according to the structured log set to obtain a serialized log set.

[0014] Step S4: pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features of the log messages to obtain a set of log embedding vectors.

[0015] Step S5: train a PCA-based classification model according to the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection.

[0016] Preferably: in step S4, the GPT-2 model is pre-trained by serializing the log set, and the method of using the trained GPT-2 model to extract features from the log messages is to train the GPT-2 model using a large number of unlabeled logs, and then use the trained GPT-2 model to extract features from the log sequence to be detected.

[0017] The objective function of the pre-trained GPT-2 model is defined as follows:

[0018]

[0019] in, represents the target loss function, N is the number of log sequences, T is the length of each sequence, Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position is, Indicates that the t+1th position is the log key k i , Represents 1 to t log keys for a given sequence i.

[0020] After pre-training, the GPT-2 model is able to generate log sequence fragments based on the given log sequence fragments. Generate a log sequence The log key of the predicted Tth position is k i , T represents the length of the sequence.

[0021] Input the serialized log S to be tested into the pre-trained GPT-2 model, obtain the last hidden layer output by the GPT-2 model, and then extract each sequence S i The vector v of the last token i , as semantic vectors, forming a log embedding vector set V, V = {v1, v2, ..., v N}, N represents the number of log sequences.

[0022] Preferred: The structure of the GPT-2 model is as follows:

[0023]

[0024] in, Indicates the use of Transformer decoder To decode, Represents the hidden layer representation derived from the Transformer decoder. Represents the hidden state The scores generated after transformation by the weight matrix W are normalized by the softmax function to obtain a probability distribution. Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position. is the head parameter of the language model, represents the real number field, d represents the length of the vector, Represents the set of all possible log keys.

[0025] Preferably, in step S5, the method of using the trained PCA-based classification model for anomaly detection is to construct a normal sample space S n and an abnormal sample space S a , then each log sequence vector v i By formula Projection to normal space S n On, where P = [e1, e2, …, e k ],e1,e2,…e krepresents the eigenvector with the first k largest eigenvalues, k is the number of principal components of the log series, v i In normal space S n Similarly, v i By formula Projection to abnormal space S a where I is the standard identity matrix. The squared prediction error is used to determine the incoming log sequence v j Whether it is an outlier in the anomaly space. j The SPE value is calculated by the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j Predicted to be an abnormal log sequence.

[0026] Preferably, the method for preprocessing the log message in step S1 is to parse the original log content according to the log format, extract the semi-structured log message, and form a log message set L, where L = {l1, l2, ..., l n}, n represents the size of the log message set, l n Represents the nth semi-structured log message.

[0027] Preferably, the specific method of structurally representing the log message in step S2 includes the following steps:

[0028] Step S2.1: downsample the log messages by a certain proportion through the hyperparameter r to form the sampled log L sample .

[0029] Step S2.2, sample log L sample Use Drain3 log parser to parse logs and extract log template set K sample , and save the unmatched log set L unmatch .

[0030] Step S2.3, again for the unmatched log set L unmatch Perform template parsing to obtain the newly extracted template, and add the newly extracted template to K sample Form a new log template set K, K = {k1, k2, ..., k m}, m represents the size of the log template set, k m Indicates the mth log template.

[0031] Step S2.4: Use the log template set K to perform template matching on the full log set L to form a structured log set D, where D = {d1, d2, …, d n}, n represents the size of the structured log set, dn Represents the nth structured log.

[0032] Preferably, the specific method of serializing the log messages in step S3 is to use a sliding time window method to group the structured log sequence into an event sequence by using the window size w and the step size s. A structured log set D with a length of n, D = {d1, d2, ..., d n}, then the i-th window can be expressed as:

[0033] s i ={d is+1 , d is+2 ,…,d (is+w)}

[0034] Here, i represents the window index, which starts from 0.

[0035] Finally, a serialized log set S is formed, S = {s1, s2, ..., s N},

[0036] Preferably, the log message includes operating system log, application log and other logs, wherein the operating system log refers to the log generated by the operating system itself. The application log refers to the log generated by the application or other related programs running on the operating system. Other logs refer to logs that do not contain abnormal information but contain other important information.

[0037] Another object of the present invention is to provide a network log anomaly detection system based on GPT2-PCA, which is used to implement the network log anomaly detection method based on GPT2-PCA, including a collection unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein:

[0038] The collection unit is used to collect log messages.

[0039] The preprocessing unit is used to preprocess the log message to obtain a log message set.

[0040] The structured log unit is used to structure the log message according to the log message set to obtain a structured log set.

[0041] The serialized log unit is used to serialize the log message according to the structured log set to obtain a serialized log set.

[0042] The GPT-2 model unit is used to pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features from the log messages to obtain a log embedding vector set.

[0043] The PCA-based classification model unit is used to train a PCA-based classification model according to the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection.

[0044] The output unit is used to output anomaly detection.

[0045] Another object of the present invention is to provide a computer system, comprising a memory and a processor, wherein the memory is used to store a computer program / instruction. The processor is used to execute the computer program / instruction to implement the network log anomaly detection method based on GPT2-PCA.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The present invention can perform fine-grained processing on a variety of logs from different devices and applications, and improve the processing speed, which is beneficial to subsequent training and testing.

[0048] (2) Using GPT-2 to extract the embedding vector of the log sequence can capture the behavioral patterns of the log sequence in a longer context.

[0049] (3) The PCA classification model is used to classify the log vectors to be detected, which has faster detection speed, higher accuracy, and is easy to implement and apply. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the overall flow chart of the present invention.

[0051] Figure 2 This is a schematic diagram of Firefox log preprocessing according to the present invention.

[0052] Figure 3 This is a schematic diagram of Windows security log preprocessing according to the present invention.

[0053] Figure 4 This is a schematic diagram of DNS log preprocessing of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0055] Example 1

[0056] A network log anomaly detection method based on GPT2-PCA, such as Figure 1 As shown, the following steps are included:

[0057] Step S1, collecting log messages. Preprocessing the log messages to obtain a log message set.

[0058] The log messages include operating system logs, application logs and other logs. The operating system logs refer to logs generated by the operating system itself. Application logs refer to logs generated by applications or other related programs running on the operating system. Other logs refer to logs that do not contain abnormal information but contain other important information.

[0059] The method for preprocessing the log message in step S1 is to parse the original log content according to the log format such as <date><time><level><source><content>, extract the semi-structured log message, and form a log message set L, L = {l1,l2,…,l n}, n represents the size of the log message set, l n Represents the nth semi-structured log message.

[0060] Step S2: According to the log message set, the log message is structured to obtain a structured log set.

[0061] The specific method of structurally representing log messages includes the following steps:

[0062] Step S2.1: downsample the log messages by a certain proportion through the hyperparameter r to form the sampled log L sample .

[0063] Step S2.2, sample log L sample Use Drain3 log parser to parse logs and extract log template set K sample , and save the unmatched log set L unmatch .

[0064] Step S2.3, again for the unmatched log set L unmatch Perform template parsing to obtain the newly extracted template, and add the newly extracted template to K sample Form a new log template set K, K = {k1, k2, ..., k m}, m represents the size of the log template set, k m Indicates the mth log template.

[0065] Step S2.4: Use the log template set K to perform template matching on the full log set L to form a structured log set D, where D = {d1, d2, …, d n}, n represents the size of the structured log set, d n Represents the nth structured log.

[0066] Step S3: Serialize the log message according to the structured log set to obtain a serialized log set.

[0067] The specific method of serializing log messages is to use the sliding time window method to group the structured log sequence into event sequences through the window size w and step size s. We have a structured log set D of length n, D = {d1, d2, ..., d n}, then the i-th window can be expressed as:

[0068] s i ={d is+1 ,d is+2 ,…,d (is+w)}

[0069] Here, i represents the window index, which starts from 0.

[0070] Finally, a serialized log set S is formed, S = {s1, s2, ..., s N},

[0071] Step S4: pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features of the log messages to obtain a set of log embedding vectors.

[0072] The method of pre-training the GPT-2 model by serializing the log collection and using the trained GPT-2 model to extract features from log messages is to train the GPT-2 model with a large number of unlabeled logs so that the GPT-2 model captures the normal behavior patterns of the log sequences, and then use the trained GPT-2 model to extract features from the log sequences to be detected.

[0073] GPT-2 is based on the Transformer decoder and uses the self-attention mechanism to capture the dependencies of log keys in the log sequence. GPT2 is trained to predict the next log key given the previous log key. The objective function of the pre-trained GPT-2 model is defined as follows:

[0074]

[0075] in, represents the target loss function, N is the number of log sequences, T is the length of each sequence, Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position is, Indicates that the t+1th position is the log key k i , Represents 1 to t log keys for a given sequence i.

[0076] To derive The structure of the GPT-2 model is as follows:

[0077]

[0078] in, Indicates the use of Transformer decoder To decode, Represents the hidden layer representation derived from the Transformer decoder. Represents the hidden state The scores generated after transformation by the weight matrix W are normalized by the softmax function to obtain a probability distribution. Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position. is the head parameter of the language model, represents the real number field, d represents the length of the vector, Represents the set of all possible log keys.

[0079] After pre-training, the GPT-2 model is able to generate log sequence fragments based on the given log sequence fragments. Generate a log sequence This means that the model has learned the normal log pattern. The log key of the predicted Tth position is k i , T represents the length of the sequence.

[0080] Input the serialized log S to be tested into the pre-trained GPT-2 model, obtain the last hidden layer output by the GPT-2 model, and then extract each sequence S i The vector v of the last token i , as a semantic vector, this is because the last token contains the entire semantics of the entire sequence, forming a log embedding vector set V, V = {v1, v2, ..., v N}, N represents the number of log sequences.

[0081] Step S5: train a PCA-based classification model based on the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection. Comprehensively judge the anomaly detection results of all logs to obtain the final attack investigation results.

[0082] The PCA-based classification model is trained according to the log embedding vector set. The method of using the trained PCA-based classification model for anomaly detection is to construct a normal sample space S n and an abnormal sample space Sa , then each log sequence vector v i By formula Projection to normal space S n On, where P = [e1, e2, …, e k ],e1,e2,…e k represents the eigenvector with the first k largest eigenvalues, k is the number of principal components of the log sequence, which also indicates that the log sequence has k principal components, called v i In normal space S n Similarly, v i By formula Projection to abnormal space S a , where I is the standard unit matrix. Since the component corresponding to the anomaly space retains the smallest variance of the original data, anomalies can be detected by outlier detection in the anomaly space. The squared prediction error is used to determine the incoming log sequence v j Whether it is an outlier in the anomaly space. j The SPE value is calculated by the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j It is predicted as an abnormal log sequence, and SPE (Square Prediction Error) represents the square prediction error.

[0083] This method can support fine-grained preprocessing of logs of multiple devices and applications, can capture normal behavior patterns of logs in a longer context, and can effectively improve the speed and accuracy of detection.

[0084] Example 2

[0085] This embodiment provides a network log anomaly detection system based on GPT2-PCA, which is used to implement the network log anomaly detection method based on GPT2-PCA, including a collection unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein:

[0086] The collection unit is used to collect log messages.

[0087] The preprocessing unit is used to preprocess the log message to obtain a log message set.

[0088] The structured log unit is used to structure the log message according to the log message set to obtain a structured log set.

[0089] The serialized log unit is used to serialize the log message according to the structured log set to obtain a serialized log set.

[0090] The GPT-2 model unit is used to pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features from the log messages to obtain a log embedding vector set.

[0091] The PCA-based classification model unit is used to train a PCA-based classification model according to the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection.

[0092] The output unit is used to output anomaly detection.

[0093] Example 3

[0094] This embodiment provides a computer system, including a memory and a processor, wherein the memory is used to store a computer program / instruction, and the processor is used to execute the computer program / instruction to implement the network log anomaly detection method based on GPT2-PCA.

[0095] Example 4

[0096] This embodiment provides a specific example of a network log anomaly detection method based on GPT2-PCA. Figure 1 As shown in the figure, consider a log anomaly detection scenario of multiple devices and multiple applications in APT attack emergency response or log audit. Logs can be divided into three categories according to their sources and functions: operating system logs, application logs, and other logs. Operating system logs refer to logs generated by the operating system itself, application logs refer to logs generated by applications or other related programs running on the operating system, and other logs refer to logs that do not contain abnormal information but contain other important information.

[0097] The overall anomaly detection is divided into three steps:

[0098] Step 1: Process the logs. The processing includes three steps: log parsing, template matching, and log grouping.

[0099] Step 2: Train the anomaly detection model. Segment the log sequence. Then use a large number of unlabeled samples to train the GPT-2 model to capture the normal pattern of the log. Finally, use the dataset to train the PCA classifier, which is an unsupervised machine learning model.

[0100] Step 3: Log anomaly detection. For the logs to be detected, first preprocess the logs, then use the trained GPT-2 model to extract features, and finally use the PCA classifier to classify and identify abnormal logs. The detection results of different logs are comprehensively judged to conduct attack investigations.

[0101] In the example of the present invention, the original log data set is from the ATLAS data set, which contains the attacked host logs. Each host collects three types of logs, Windows security logs, Firefox browser logs, and DNS resolution logs, which belong to operating system logs, application logs, and other logs respectively.

[0102] Step 1: Log preprocessing. This dataset provides logs of three different applications. The log preprocessing method can be adjusted according to the different log formats. Figure 2 , 3 ,4 respectively show the preprocessing methods and results for three different logs.

[0103] Figure 2 This article shows how to preprocess Firefox logs. <date> <time> <utc> -[ <thread> ]: <level> / <component> <content>, parse the original log and extract the semi-structured log message, that is, <content>part, forming a log message set L, L = {l1,l2,…,l n }. The log message L is conditioned by the hyperparameter r. Firefox Perform a certain proportion of downsampling to form L sample . For the sampled log L sample Use Drain3 log parser to parse logs and extract log template set K sample , and save the unmatched log set L unmatch . Again for the unmatched log set L unmatch Perform template parsing and add the newly extracted template to K sample , forming a new log template set K, K = {k1, k2, ..., k m }. Use the log template set K to perform template matching on the full log set L to form a structured log set D, where D = {d1, d2, …, d n Using the sliding time window technique, the structured log sequence is grouped into event sequences by window size w and step size s. The i-th window can be expressed as: s i ={d is+1 ,d is+2 ,…,d (is+w) }, where i represents the window index, starting from 0. Finally, the serialized log set S is formed Firefox , S firefox ={s1,s2,…,s N },

[0104] Figure 3 This paper shows the preprocessing method for Windows security log. WinSec ={l1,l2,…,l n }, extract the event ID of each log message, which is a number used by Windows security logs to identify specific event types. Each number represents a different log event, forming a structured log set D, D = {d1, d2, ..., d n The logs generated by the same process are grouped into a serialized log set S, where S = {s1, s2, …, s k }, where k is the number of different processes in the log message set, s i ={d1,d2,…,d m }, m is the number of logs generated by the process. Use the sliding window technology to trim overlong log sequences. By specifying the time window size w, log sequences s with a length greater than w are trimmed. i Crop to sub-windows. Finally, the serialized log set S is formed. WinSec , S WinSec ={d1,d2,…,d N }.

[0105] Figure 4 This paper shows the preprocessing method of DNS resolution log. The correspondence between domain names and IP addresses in DNS log is saved using a dictionary and converted into JSON format and saved in a file.

[0106] Step 2: Train the anomaly detection model. Use a large number of unlabeled logs to train the GPT-2 model so that GPT-2 can capture the behavior patterns of benign logs, and then use the GPT-2 model to extract features from the log sequences to be detected.

[0107] GPT-2 is based on the Transformer decoder and uses the self-attention mechanism to capture the dependencies of log keys in the log sequence. GPT-2 is trained to predict the next log key given the previous log key. The objective function of pre-training GPT-2 is defined as follows:

[0108]

[0109] Where N is the number of log sequences, T is the length of each sequence, Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position.

[0110] To derive The structure of GPT2 is as follows:

[0111]

[0112]

[0113] in represents the hidden representation obtained from the Transformer decoder, and is the head parameter of the language model, which maps the hidden representation to all log keys The probability distribution of .

[0114] After pre-training, GPT-2 is able to generate a log sequence based on a given segment. Generate a log sequence This means that the model has learned the pattern of normal logs.

[0115] In this example, GPT-2 is trained using unlabeled Firefox logs and Windows security logs. After training, the trained GPT-2 model is used to extract features from these benign logs. The serialized log S is input into the model, the last hidden layer of the model output is obtained, and then each sequence S is extracted. i The vector v of the last token i , as a semantic vector, this is because the last token contains the entire semantics of the entire sequence. Form a log embedding vector set V, V = {v1, v2, ..., v N }.

[0116] Then use the log embedding vector set V to train the PCA-based classification model. Construct a normal sample space (using S n ), and an abnormal sample space (denoted by S a ), and then each log sequence vector (denoted by v i Represented by the formula Projection into normal space Above. Where P = [e1, e2, …, e k ], and e1,e2,…e k represents the eigenvector with the largest eigenvalue. This also shows that the log series has k principal components. v i In normal space S n Similarly, v i By formula Projection to abnormal space S a , where I is the standard unit matrix. Since the component corresponding to the abnormal space retains the minimum variance of the original data, the anomaly can be detected by outlier detection in the abnormal space.

[0117] Step 3: Log anomaly detection. Perform log preprocessing on the log to be detected through step 1 to form a log sequence S′ to be detected. Use the pre-trained GPT-2 model to extract features from the log sequence to be detected through step 2 to form an embedding vector V′. Use the PCA-based classification model trained in step 2 for classification. Determine the incoming log sequence by squared prediction error (using v j Indicates whether v is an outlier in the anomaly space. j The SPE value is calculated by the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j Predicted to be an abnormal log sequence.

[0118] The obtained abnormal log sequence is summarized, and the log ID is mapped with the structured log message D to extract the entities such as URL, process, etc. in the malicious structured log parameter list to obtain the malicious entity. Through the corresponding relationship in the DNS log, the connection between entities is discovered, and a comprehensive attack investigation is conducted, while eliminating false positive information.

[0119] The present invention proposes a fine-grained preprocessing method for multiple logs of different devices and applications, and improves the processing speed, solving the problem of excessive granularity of existing log preprocessing methods, which is beneficial to subsequent training and detection. A method for extracting log sequence embedding vectors using GPT-2 is proposed, which can capture the behavior pattern of log sequences in a longer context, solving the problem that existing feature extraction methods have weak ability to capture anomalies in sequences. A method for classifying log vectors to be detected using a PCA classification model is proposed, which has a fast classification speed and higher accuracy, and is easy to implement and apply, solving the problems of slow classification speed and low classification performance of existing classification models.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / content> < / content> < / component> < / level> < / thread> < / utc> < / time> < / date>

Claims

1. A network log anomaly detection method based on GPT2-PCA, characterized in that: The following steps are involved: Step S1, collecting log messages; preprocessing the log messages to obtain a log message set; Step S2: According to the log message set, the log message is structured to obtain a structured log set; Step S3: Serializing the log message according to the structured log set to obtain a serialized log set; Step S4: pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features from the log messages to obtain a set of log embedding vectors; Step S5: train a PCA-based classification model according to the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection.

2. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: In step S4, the GPT-2 model is pre-trained by serializing the log set, and the method of using the trained GPT-2 model to extract features from the log message is to train the GPT-2 model using a large number of unlabeled logs, and then use the trained GPT-2 model to extract features from the log sequence to be detected; The objective function of the pre-trained GPT-2 model is defined as follows: in, represents the target loss function, N is the number of log sequences, T is the length of each sequence, Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position is, Indicates that the t+1th position is the log key k i , represents 1 to t log keys for a given sequence i; After pre-training, the GPT-2 model is able to generate log sequence fragments based on the given log sequence fragments. Generate a log sequence The log key of the predicted Tth position is k i , T represents the length of the sequence; Input the serialized log S to be tested into the pre-trained GPT-2 model, obtain the last hidden layer output by the GPT-2 model, and then extract each sequence S i The vector v of the last token i , as semantic vectors, forming a log embedding vector set V, V = {v1, v2, ..., v N }, N represents the number of log sequences.

3. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: The structure of the GPT-2 model is as follows: in, Indicates the use of Transformer decoder To decode, represents the hidden layer representation derived from the Transformer decoder, Represents the hidden state The scores generated after transformation by the weight matrix W are normalized by the softmax function to obtain a probability distribution. Indicates that in a given sequence In the case of , the probability of the log key predicted by GPT-2 at the t+1th position is, is the head parameter of the language model, represents the real number field, d represents the length of the vector, Represents the set of all possible log keys.

4. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: The method of using the trained PCA-based classification model for anomaly detection in step S5 is to construct a normal sample space S n and an abnormal sample space S a , then each log sequence vector v i By formula Projection to normal space S n On, where P = [e1, e2, …, e k ],e1,e2,…e k represents the eigenvector with the first k largest eigenvalues, k is the number of principal components of the log series, v i In normal space S n The projection on i By formula Projection to abnormal space S a where I is the standard identity matrix; the squared prediction error is used to determine the incoming log sequence v j Is it an outlier in the anomaly space? j The SPE value is calculated by the formula To calculate, by comparing with a given threshold θ, if the SPE value is greater than θ, then v j Predicted to be an abnormal log sequence.

5. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: The method for preprocessing the log message in step S1 is to parse the original log content according to the log format, extract the semi-structured log message, and form a log message set L, where L = {l1, l2, ..., l n }, n represents the size of the log message set, l n Represents the nth semi-structured log message.

6. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: The specific method of performing structured representation of the log message in step S2 includes the following steps: Step S2.1: downsample the log messages by a certain proportion through the hyperparameter r to form the sampled log L sample ; Step S2.2, sample log L sample Use Drain3 log parser to parse logs and extract log template set K sample , and save the unmatched log set L unmatch ; Step S2.3, again for the unmatched log set L unmatch Perform template parsing to obtain the newly extracted template, and add the newly extracted template to K sample Form a new log template set K, K = {k1, k2, ..., k m }, m represents the size of the log template set, k m Indicates the mth log template; Step S2.4: Use the log template set K to perform template matching on the full log set L to form a structured log set D, where D = {d1, d2, …, d n }, n represents the size of the structured log set, d n Represents the nth structured log.

7. The network log anomaly detection method based on GPT2-PCA according to claim 1 is characterized in that: The specific method of serializing the log messages in step S3 is to use a sliding time window method to group the structured log sequence into an event sequence through the window size w and the step size s; a structured log set D with a length of n, D = {d1, d2, ..., d n }, then the i-th window can be expressed as: s i ={d is+1 ,d is+2 ,…,d (is+w) } Among them, i represents the window index, i starts from 0; Finally, a serialized log set S is formed, S = {s1, s2, ..., s N }, 8. The network log anomaly detection method based on GPT2-PCA according to claim 1, characterized in that: The log messages include operating system logs, application logs and other logs. The operating system logs refer to logs generated by the operating system itself; application logs refer to logs generated by applications or other related programs running on the operating system; other logs refer to logs that do not contain abnormal information but contain other important information.

9. A network log anomaly detection system based on GPT2-PCA, characterized in that: A network log anomaly detection method based on GPT2-PCA for implementing any one of claims 1 to 6, comprising a collection unit, a preprocessing unit, a structured log unit, a serialized log unit, a GPT-2 model unit, a PCA-based classification model unit, and an output unit, wherein: The collection unit is used to collect log messages; The preprocessing unit is used to preprocess the log message to obtain a log message set; The structured log unit is used to perform structured representation on the log message according to the log message set to obtain a structured log set; The serialized log unit is used to serialize the log message according to the structured log set to obtain a serialized log set; The GPT-2 model unit is used to pre-train the GPT-2 model by serializing the log set, and use the trained GPT-2 model to extract features from the log message to obtain a log embedding vector set; The PCA-based classification model unit is used to train a PCA-based classification model according to the log embedding vector set, and use the trained PCA-based classification model to perform anomaly detection; The output unit is used to output anomaly detection.

10. A computer system, characterized in that: It includes a memory and a processor, wherein the memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions to implement the network log anomaly detection method based on GPT2-PCA as described in any one of claims 1-8.

Images