Multi-step sustainability attack detection method based on multi-source logs

Abstract

The application provides a multi-step sustainability attack detection method based on multi-source logs, and mainly solves the problem of poor prediction accuracy of existing malicious multi-step sustainability attack behaviors. 1) Collect system logs, application logs, network logs and other multi-source logs for scoring and clustering; 2) Perform three-dimensional tensor modeling on the logs, and utilize tensor decomposition to extract events through layer-by-layer decomposition of the log data; 3) Construct an event traceability graph according to the event relationship and extract an event path; 4) Model the causal relationship of the events; 5) Model the neighborhood relationship of the events by utilizing a Poisson-Gamma mixed distribution; 6) Establish a factor graph model to predict the attack stage of an attacker, and obtain a stage relationship matrix; 7) Input three relationship matrices into a classifier combined by an LSTM and a BNN to perform malicious attack event path prediction. The application can realize multi-step sustainability attack detection by utilizing original multi-source logs, and effectively improves the detection accuracy.

Description

Technical Field

[0001] This invention belongs to the field of information security technology, and further relates to attack prediction methods, specifically a multi-step sustainable attack detection method based on multi-source logs, which can be used to predict malicious multi-step sustainable attack behavior in daily activities. Background Technology

[0002] Multi-step persistence attacks refer to targeted cyberattacks launched by an organization against a specific target over a prolonged period. These attacks are characterized by high complexity and stealth, long duration, and potentially significant damage. Due to the complexity and long-term behavior of multi-step persistence attacks, traditional threat detection systems are often insufficient for modeling and detecting them. Traditional network prediction methods and attack detection systems are unsuitable for modeling long-running multi-step persistence attack behavior and perform poorly in predicting advanced attack behaviors such as zero-day vulnerabilities. Furthermore, most existing solutions are based on readily available event flow graphs, which are inapplicable to multi-source native logs. Therefore, techniques for modeling the long-term behavior of multi-step persistence attacks using complex multi-source logs have become a hot research topic for scholars both domestically and internationally.

[0003] The current framework for predicting and proactively defending against multi-step persistence attacks generally consists of three main steps. First, a multi-step persistence attack event source graph is constructed. This is done by using information such as multi-step persistence attack behavior reports, system logs, and network logs to construct multi-step persistence attack events and then creating an event graph based on the relationships between these events. Next, the relationships between these multi-step persistence attack events are modeled. To translate these relationships into input requirements for a neural network, various behaviors of multi-step persistence attack events are modeled, capturing the relationships between attacks and predicting future attacker behavior. Finally, in the training phase, the relationship matrix is ​​fed into a suitable neural network to predict malicious behavior sequences and take proactive action for defense.

[0004] Multi-step persistence attacks are more difficult to defend against than ordinary network defenses due to their long duration and sophisticated attack strategies. Current defense methods, aiming to accurately predict attacker behavior and block attacks before they occur, employ signature-based and anomaly-based detection schemes to predict all possible attacker actions and utilize machine learning to learn from historical attacker data and related information to detect malicious events and prevent attacks. However, these approaches and existing technologies suffer from the following drawbacks. First, attackers leverage the massive volume, diverse sources, and low value density of network data to disperse attacks across various types and sources of network behavior over a long period. This makes anomaly analysis methods for short-cycle traditional network threats less applicable to multi-step persistence attacks. Second, traditional threat detection systems are unsuitable for detecting long-running multi-step persistence attack activities, and systems based on the aforementioned methods perform poorly against advanced attack techniques such as zero-day vulnerabilities. Furthermore, existing anomaly detection systems cannot model long-running network behavior; poor relationship modeling results in low prediction accuracy and high memory consumption.

[0005] Fan Shen et al. published a host-based multistage APT detection and response model, SR2APT, in their paper "SR2APT: A Detection and Strategic Alert Response Model against Multistage APT Attacks" (Security and Communication Networks, 2023). This framework, based on graph convolutional networks, classifies the subgraph stream of system log sources into either multistage APT attack stages or benign stages. The detection results are then sequentially transmitted to a decision engine, which is trained using deep reinforcement learning to output the optimal response action for multistage APT alerts. However, this method suffers from a single target data source, and the classification of nodes based on attack stages is not a perfect metric, leading to low detection accuracy. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the aforementioned technologies and provide a method for detecting multi-step sustainable attacks based on multi-source logs. This method primarily addresses the problem of predicting malicious multi-step sustainable attack behaviors from daily activities using heterogeneous multi-source data. First, considering the heterogeneity, complexity, unstructured nature, and diverse sources of data, this invention designs a multi-step sustainable attack event extraction scheme based on tensor decomposition. Tensor decomposition is used to extract templates from complex and diverse logs, and then the spatiotemporal patterns of events are extracted based on these templates, with the resulting data being extracted as events. Second, a multi-step sustainable attack relationship modeling method is proposed, used to model and predict multi-step sustainable attack behaviors using event graphs generated from multi-source logs. This method can model the causal relationships, neighborhood relationships, and stage relationships between processes and objects in the system, predicting the attacker's current stage and potential attack behaviors. Furthermore, it can provide low-cost defense strategies to achieve accurate and economical proactive defense.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Collect system logs, application logs and network logs, and use regular expression matching to parse the log statements into corresponding structured entities to obtain structured logs;

[0009] (2) Extract the log template from the structured log, as follows:

[0010] (2.1) First, perform word scoring. Assume that the words in each log entry are separated by spaces. Use spaces to segment the words and record the position p of each word and the length len of the log message during the segmentation process. Set the scoring rule as follows: if a word appears at the p-th position of a log message of length len, then the score of this word can be calculated according to the following formula:

[0011] Score(word,p,len)=p(word|p,len);

[0012] (2.2) Divide all words in each message into template words and parameter words. The template words are log entries with the same length that appear in the same position. Classify the words according to the score obtained in (2.1) and use the density-based spatial clustering DBSCAN method to cluster them so that the distance between clusters is greater than the distance threshold δ. Then, select the cluster with the highest score so that the number of words is greater than β×Len, where β is the parameter word count threshold and 0<β<1, to obtain the log template.

[0013] (3) Treat network log data as a rank 3 tensor, namely log template, event and time window, and extract the template group and host relationship that appear at the same time in the log data, as follows;

[0014] (3.1) The log data for each time window is decomposed layer by layer, as follows:

[0015] (3.1.1) Define a log template matrix V of size I×L. If template i belongs to template group l, then the template components v of the log template matrix are... il >0, if template i does not belong to template group l, then it equals v il =0;

[0016] (3.1.2) Let Z be a rank-3 tensor to represent an event tensor. Each H×L matrix is ​​an event slice. In each event slice, if template group l on host h belongs to the k-th event, then the event component z corresponding to that event is... lkh >0; conversely, z lkh =0;

[0017] (3.1.3) Let W denote the time window weight matrix, and represent the weight component of the k-th event occurring in the j-th time window as w. kj This completes the layer-by-layer decomposition of the log tensor, obtaining the components of log template, event, and time window weights;

[0018] (3.2) Represent the decomposition problem as a minimization problem defined by KL divergence and loss function, iteratively calculate the components of the log tensor according to the multiplicative update rule, and extract the events from it;

[0019] (4) Represent each internal node in the current log as a process creation event, and the child nodes of the internal nodes as events of process-system resource interaction. Connect the events to form an event sourcing graph.

[0020] (5) Extract and store the event paths based on the event source graph using a depth-first traversal method;

[0021] (6) Using the extracted event paths, model the causal relationships of the events and construct a causal relationship matrix:

[0022] (6.1) Define the relationship in the following four dimensions:

[0023] Dimension 1: The difference in timestamps between the parent and child events;

[0024] Dimension 2: The process ID of the parent event, the parent process ID of the parent event, the host ID of the parent event, and the process name of the parent process;

[0025] Dimension 3: The IP address and port of the parent event;

[0026] Dimension 4: The type of the parent event;

[0027] (6.2) For each event path extracted in step (5), extract the four-dimensional relationships defined in step (6.1) and construct the corresponding causal relationship matrix; each event in the matrix corresponds to a one-dimensional row vector of the causal relationship, and the size of the matrix is ​​L×8, where L represents the path length;

[0028] (7) Model the neighborhood relationships of events and construct a neighborhood relationship matrix:

[0029] (7.1) Use the first N columns of the neighborhood matrix to represent the deviation between the past and present, where N is the number of event types, and obtain the event deviation vector D = {D1, D2, ..., D...}. N};

[0030] (7.2) Use the Poisson-Gamma mixture distribution to predict the future neighborhood relationships of events, calculate the probability of each type of event occurring after a unit of time, and store the probabilities using a vector P, where P = {P1, P2, ..., P...} N};

[0031] (7.3) Connecting vectors D and P forms a neighborhood matrix, where each event in the matrix corresponds to a one-dimensional row vector of neighborhood relations, and the size of the matrix is ​​L×16;

[0032] (8) Model the stage relationships of the event and construct a stage relationship matrix:

[0033] (8.1) Using four types of variables as input variables—user information, events occurring on the user's node, the attack stage the user is in, and the attack node the user is on—defin the following four factor functions and capture the relationships between these input variables:

[0034] The first type: the attack phase s of a multi-step sustainable attack at time t. t With the user in the current information event e t The relationship between f e (e t ,s t If event e t If this occurs, it is believed that multi-step sustainability attacks currently exist in s t This stage;

[0035] The second type: the attack phase s of a multi-step sustainable attack at time t. t With the current event e t Attack phase s at time t-1 t-1 Attack event e t-1 The relationship between the three s (e t-1 ,s t-1 ,e t ,s tIf event e t An attack event occurred, and at time t-1, an attack event e occurred. t-1 And time t-1 is located at s t-1 The phase, however, suggests that multi-step sustainability attacks currently exist in s t This stage;

[0036] The third type: the attack phase s of a multi-step sustainable attack at time t. t With user attribute U, the currently occurring event e t The relationship between the two f u (U,e t ,s t If event e t If an attack occurs and the user has attribute U, then a multi-step persistence attack is considered to currently exist in s. t This stage;

[0037] The fourth type: the attack phase s of a multi-step sustainable attack at time t. t The attack node (Node) where the user is located, and the current event (e) t The relationship between the two f n (Node,e t ,s t If event e t If an attack occurs and the user is on the attacking node, then a multi-step sustainability attack is considered to currently exist on node s. t This stage;

[0038] (8.2) Combine all factor functions into a set F = {F e ,F s ,F u ,F n}, where F e F s F u F n f in the factor graph e (e t ,s t ), f s (e t-1 ,s t-1 ,e t ,s t ), f u (U,e t ,s t ) and f n (Node,e t ,s t The set of factors; let the input of each factor function be X = {e t ,U,Node}、Y={S tThe conditional probability density P(Y|X) is calculated when X and Y satisfy the correlation relationship.

[0039] (8.3) Construct a matrix of size L×7 using conditional probability. Each event in the matrix corresponds to a one-dimensional row vector of length 7 in the neighborhood relationship, representing the seven attack stages of a multi-step sustainable attack. Set the value of the column corresponding to the stage most likely to be in each row to P, and set the rest of the columns to 0 to obtain the stage relationship matrix.

[0040] (9) For each event path, input its corresponding causal relationship matrix, neighborhood relationship matrix and stage relationship matrix into a three-layer long short-term memory neural network LSTM and a one-layer Bayesian neural network BNN, and train the classifier using ELBO loss;

[0041] (10) The trained classifier is used to predict the event path and generate the prediction results of the category, indicating whether the event is benign or a multi-step sustainable attack. At the same time, an uncertainty score representing the standard deviation of the prediction is generated to realize the detection of multi-step sustainable attacks.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] First, because this invention uses network behavior extraction based on tensor decomposition of multi-source heterogeneous data to extract events, by modeling the three-dimensional tensor of log data and extracting events by decomposing the log data layer by layer, the source graph can be constructed using native multi-source logs to construct event sequences. Compared with existing technologies that require specific event datasets, network logs are used in a single way, and the data input requirements are more universal, requiring only native logs and being compatible with various types of logs, such as network logs and system logs. This multi-source heterogeneous event extraction method makes the applicability of this invention stronger, and considering multiple types of logs can also increase the collection of information from data sources, thereby improving the accuracy of prediction.

[0044] Secondly, this invention employs causal and neighborhood relationship modeling methods. It models causal relationships through multi-dimensional relationships and neighborhood relationships through a Poisson-Gamma distribution, making it suitable for modeling long-lasting, multi-step sustainable attack actions. Existing methods rely on traditional intrusion detection systems or ordinary network behavior anomaly detection schemes, which are difficult to adapt to detecting long-lasting, multi-step sustainable attacks, resulting in low prediction accuracy. Alternatively, simple causal and neighborhood modeling methods are not entirely suitable for multi-step sustainable attacks. For example, while using a Poisson distribution to predict multi-step sustainable attacks, these attacks exhibit temporal variability, and a constant event probability is unsuitable for modeling them. This invention utilizes a Poisson-Gamma distribution to model event occurrence rates, improving the accuracy of probability prediction. Therefore, the modeling scheme of this invention is adjusted according to the characteristics of multi-step sustainable attacks, resulting in better adaptability and higher accuracy in modeling such attacks.

[0045] Third, this invention adds a stage-based modeling method. By using a factor graph-based modeling approach, it predicts the multi-step sustainability attack stage of a newly occurring event, aiming to preemptively prevent multi-step sustainability attacks before they occur. Existing solutions use convolutional neural networks (CNNs) to divide multi-step sustainability attacks into stages, considering only historical attack datasets, which limits scalability and accuracy. The factor function in this invention can be dynamically adjusted, setting the most suitable factor function based on specific multi-step sustainability attack behaviors. Furthermore, this solution simultaneously considers multi-faceted information from current events, past events, and user information, resulting in a more comprehensive and accurate prediction of attack stages compared to multi-step sustainability attack stage division schemes that only consider historical datasets.

[0046] Fourth, this invention employs a temporal modeling approach combining a Long Short-Term Memory (LSTM) neural network and a single-layer Bayesian Neural Network (BNN), using ELBO loss to train the classifier. Compared to existing solutions that use RNNs or LSTMs for modeling and prediction, this method increases the confidence level of the prediction results. That is, while a regular neural network will still select the most suitable prediction even for samples with all incorrect predictions, the addition of the Bayesian neural network provides an uncertainty score during prediction, preventing the blind assignment of inappropriate predictions to uncertain samples. Therefore, this classifier achieves higher prediction accuracy with data containing a large number of normal activity samples. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the tensor decomposition process in this invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0050] Example 1: Refer to Figure 1 This invention proposes an APT attack prediction method based on Bayesian attack graphs and factor graphs, which specifically includes the following steps:

[0051] Step 1. Collect system logs, application logs, and network logs, and parse them using regular expression matching to convert the log statements into corresponding structured entities, thus obtaining structured logs;

[0052] Step 2. Extract the log template from the structured log, as follows:

[0053] Given the unstructured and complex nature of logs, a log template is first extracted from the raw logs, and then information related to network events is extracted from the template.

[0054] (2.1) First, word scoring is performed. Assuming words in each log entry are separated by spaces, words are segmented using spaces, thus dividing them into two categories: template words and parameter words. During the segmentation process, the word position p and the log message length len are recorded. The probability of each word being a template word is calculated using a conditional probability formula, and this probability is used as the word's score. The scoring rule is set as follows: if a word appears at the p-th position in a log message of length len, then the word's score can be calculated according to the following formula:

[0055] Score(word,p,len)=p(word|p,len);

[0056] (2.2) Next, score clustering is performed. It is necessary to determine whether the words are parameter words or template words. All words in each message are divided into template words and parameter words. The template words are log entries with the same length that appear in the same position. The words are classified according to the scores obtained in (2.1). Density-based spatial clustering DBSCAN is used to cluster the words so that the distance between the clusters is greater than the distance threshold δ. Then, a template is obtained by selecting the highest-scoring cluster, i.e., the top cluster, so that the number of words is greater than β×Len, where β is the parameter word count threshold and 0<β<1. The log template is obtained.

[0057] Step 3. After defining the log templates, we can use them to define network events. The core idea here is to treat network log data as a rank-3 tensor (log template, event, and time window) and extract template groups and host relationships that may appear simultaneously in the log data. A non-negative tensor decomposition method is used, allowing us to directly model and learn template groups and network events.

[0058] Treating network log data as a rank-3 tensor, namely log template, event, and time window, we extract the template group and host relationship that appear simultaneously in the log data, as follows:

[0059] (3.1) Log data is decomposed layer by layer.

[0060] The key is to view the log data for each time window as a layered overlay; that is, the log data is a mixture of network events, including template groups. The decomposition of template groups refers to... Figure 2 The first step in the layer-by-layer decomposition is the decomposition of the template group, followed by the decomposition of the network event layer, and finally represented by the weight matrix; this completes the layer-by-layer decomposition of the log tensor, transforming it into components of template groups, network events, and weights. The layer-by-layer decomposition of log data for each time window follows these steps:

[0061] (3.1.1) Define a log template matrix V of size I×L. If template i belongs to template group l, then the template components v of the log template matrix are... il >0, if template i does not belong to template group l, then it equals v il =0;

[0062] (3.1.2) Let Z be a rank-3 tensor to represent an event tensor. Each H×L matrix is ​​an event slice. In each event slice, if template group l on host h belongs to the k-th event, then the event component z corresponding to that event is... lkh >0; conversely, z lkh =0;

[0063] (3.1.3) Let W denote the time window weight matrix, and represent the weight component of the k-th event occurring in the j-th time window as w. kj This completes the layer-by-layer decomposition of the log tensor, obtaining the components of log template, event, and time window weights;

[0064] (3.2) Represent the decomposition problem as a minimization problem defined by KL divergence and loss function, iteratively calculate the components of the log tensor according to the multiplicative update rule, and extract the events from it;

[0065] The multiplication update rule is defined by the following formula:

[0066]

[0067] in, These represent the component values ​​corresponding to the iterative weights of the old log template, event, and time window, respectively. The template and parameter terms are processed using a hierarchical decomposition of the log. The decomposed variables are then substituted into the update rule formula for iterative calculation to obtain the event corresponding to the current log. The calculation results are used to analyze the network event situation on the target host and serve as event data to aid subsequent work.

[0068] Step 4. After acquiring the event, each internal node in the event corresponding to the current log is represented as a process creation event. The child nodes of the internal node are represented as events of process interaction with system resources, such as opening / modifying / deleting files, sending data packets over the network, starting or communicating with another process, etc. The event sourcing graph is formed by connecting the events.

[0069] Step 5. Based on the event source graph, extract and store the event paths using a depth-first traversal. Specifically: First, initialize a stack to store the event paths during the depth-first traversal. Then, start a depth-first traversal from a leaf node, visiting child nodes downwards and pushing visited child nodes onto the stack until the root node is reached. Record that the nodes in the stack at this point constitute an event path. Begin popping from the stack and backtracking, retracing unvisited child nodes, repeating the event path recording process until all nodes in the graph have been traversed, and storing all event paths.

[0070] Step 6. Using the extracted event paths, model the causal relationships of the events and construct a causal relationship matrix:

[0071] (6.1) Define the relationship in the following four dimensions:

[0072] Dimension 1: The difference in timestamps between the parent and child events;

[0073] Events within the same period are often interconnected, so we use the time difference between child and parent events to establish the connections between relationships in this dimension.

[0074] Dimension 2: The process ID of the parent event, the parent process ID of the parent event, the host ID of the parent event, and the process name of the parent process;

[0075] The parent event's process ID, the parent event's parent process ID, the parent event's host ID, and the parent process's process name. Malicious events' parent processes are often also malicious, so we use IDs and process names to establish connections in this dimension. We added host information to this dimension to track process IDs, because a process has two different process IDs on two different hosts, while the process name is permanent.

[0076] Dimension 3: The IP address and port of the parent event;

[0077] The IP address and port of the parent event. Through analysis of network packets during the attack, we discovered that malicious events in multi-step sustainable attacks have related IP and port information.

[0078] Dimension 4: The type of the parent event;

[0079] Event types can better help us capture the causal relationships, neighborhood relationships, and stage relationships of events. Therefore, we use event types to set the connections of relationships in this dimension.

[0080] (6.2) For each event path extracted in step (5), extract the four-dimensional relationships defined in step (6.1) and construct the corresponding causal relationship matrix; each event in the matrix corresponds to a one-dimensional row vector of the causal relationship, and the size of the matrix is ​​L×8, where L represents the path length;

[0081] Step 7. Model the neighborhood relationships of events and construct a neighborhood relationship matrix:

[0082] Modeling the neighborhood relationships of events involves using these relationships to model the past and future properties of the events. The past refers to the properties of events that occurred before the event, while the future uses the distribution of events to predict the probability of future events.

[0083] (7.1) Use the first N columns of the neighborhood matrix to represent the deviation between the past and present, where N is the number of event types, and obtain the event deviation vector D = {D1, D2, ..., D...}. N This embodiment uses four event types as examples, assuming N=4, with four columns: process, file, flow, and shell. The discrepancies are stored using a deviation vector D, where D = {D...} process D file D flow D shell The calculation method is as follows:

[0084]

[0085] Where timediff refers to the time elapsed from the initial event to the current event, and Avg(type) refers to the average probability of an event of type type occurring, calculated as follows:

[0086]

[0087] Where sumTime represents the time interval from the initial event to the current new event, and sumNum(type) represents the total number of events of type type that occur within the sumTime time range.

[0088] (7.2) Use the Poisson-Gamma mixture distribution to predict the future neighborhood relationships of events, calculate the probability of each type of event occurring after a unit of time, and store the probabilities using a vector P, where P = {P1, P2, ..., P...} N}

[0089] Here, we calculate the probability of each type of event occurring after a unit of time, which is equivalent to calculating the probability of a Poisson-Gamma mixture distribution:

[0090] The probability density function of the gamma distribution is:

[0091]

[0092] Where k and θ are the shape parameters of the gamma distribution, Γ(k) represents the gamma function, and λ represents the average number of events occurring per unit time.

[0093] The probability mass function of the Poisson distribution is:

[0094]

[0095] Where k represents the number of times a specific event occurs;

[0096] The probability mass function of the Poisson-Gamma mixture distribution is expressed as follows:

[0097]

[0098] (7.3) Connecting vectors D and P forms a neighborhood matrix, where each event in the matrix corresponds to a one-dimensional row vector of neighborhood relations, and the size of the matrix is ​​L×16;

[0099] Step 8. Model the stage relationships of the event and construct a stage relationship matrix:

[0100] At this point, modeling the stage relationships of the event is proposed, namely, a stage division method for attack paths based on factor graphs. A factor graph is a probabilistic graphical model composed of random variables and factor functions. In a factor graph, a random variable node refers to the user's behavior observed based on the event and the corresponding multi-step sustainable attack stage. Random variable nodes are connected by factor functions that represent the causal relationships between random variables.

[0101] (8.1) Using four types of variables as input variables—user information, events occurring on the user's node, the attack stage the user is in, and the attack node the user is on—defin the following four factor functions and capture the relationships between these input variables:

[0102] The first type: the attack phase s of a multi-step sustainable attack at time t. t With the user in the current information event e tThe relationship between f e (e t ,s t If event e t If this occurs, it is believed that multi-step sustainability attacks currently exist in s t This stage;

[0103] The second type: the attack phase s of a multi-step sustainable attack at time t. t With the current event e t Attack phase s at time t-1 t-1 Attack event e t-1 The relationship between the three s (e t-1 ,s t-1 ,e t ,s t If event e t An attack event occurred, and at time t-1, an attack event e occurred. t-1 And time t-1 is located at s t-1 The phase, however, suggests that multi-step sustainability attacks currently exist in s t This stage;

[0104] The third type: the attack phase s of a multi-step sustainable attack at time t. t With user attribute U, the currently occurring event e t The relationship between the two f u (U,e t ,s t If event e t If an attack occurs and the user has attribute U, then a multi-step persistence attack is considered to currently exist in s. t This stage;

[0105] The fourth type: the attack phase s of a multi-step sustainable attack at time t. t The attack node (Node) where the user is located, and the current event (e) t The relationship between the two f n (Node,e t ,s t If event e t If an attack occurs and the user is on the attacking node, then a multi-step sustainability attack is considered to currently exist on node s. t This stage;

[0106] (8.2) Combine all factor functions in the complete factor graph into a set F = {F e ,F s ,F u ,F n}, where F e F sF u F n f in the factor graph e (e t ,s t ), f s (e t-1 ,s t-1 ,e t ,s t ), f u (U,e t ,s t ) and f n (Node,e t ,s t The set of factors; let the input of each factor function be X = {e t ,U,Node}、Y={S t The conditional probability density P(Y|X) is calculated as follows: When X and Y satisfy the correlation relationship, output 1; otherwise, output 0.

[0107] The joint probability density P(E) of X and Y is obtained according to the following formula. t ,S t (U,N):

[0108]

[0109] Calculate the conditional probability density P(Y|X):

[0110]

[0111] Where Z is the normalization factor and Π represents the cumulative multiplication symbol.

[0112] (8.3) We construct the third stage information encoding matrix using St. In simple terms, St represents two elements: the stage most likely each node is in on the event path and the probability of being in that combination. Therefore, we construct an L×7 matrix using conditional probabilities. Each event in the matrix corresponds to a one-dimensional row vector of length 7 representing the seven attack stages of a multi-step sustainable attack. We set the value of the column corresponding to the most likely stage in each row to P, and set the remaining columns to 0, thus obtaining the stage relation matrix.

[0113] Step 9. For each event path, input its corresponding causal relationship matrix, neighborhood relationship matrix, and stage relationship matrix, namely the L×8 matrix formed by causal relationship, the L×16 matrix formed by neighborhood relationship, and the L×7 matrix formed by stage relationship, into a three-layer long short-term memory neural network LSTM and a one-layer Bayesian neural network BNN, and train the classifier using ELBO loss.

[0114] Step 10. Use the trained classifier to predict the event path and generate prediction results for the category, indicating whether the event is benign or a multi-step sustainable attack. At the same time, generate an uncertainty score representing the standard deviation of the prediction. We mark multi-step sustainable attack events with high certainty scores as malicious behavior and carry out proactive defense, ultimately achieving multi-step sustainable attack detection.

[0115] The technical effects of the present invention will be further explained below with reference to simulation results:

[0116] The above simulation analysis proves the correctness and effectiveness of the method proposed in this invention.

[0117] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A multi-step sustainability attack detection method based on multi-source logs, characterized in that, Comprising the following steps: (1) Collect system logs, application logs and network logs, and parse them into structured entities using regular matching, to obtain structured logs; (2) Extract log templates from structured logs, as follows: (2.1) First, score the words, assuming that the words in each log entry are separated by spaces, and use the spaces to tokenize, and record the position p and the length len of the log message during tokenization; Set the scoring rule as follows: if a word appears at the p-th position of a log message with length len, then the score of the word can be calculated according to the following formula: Score(word, p, len) = p(word|p, len); (2.2) Divide all the words in each message into template words and parameter words, wherein the template words are log entries with the same length appearing at the same position; Classify the words according to the score obtained in (2.1), and use the DBSCAN method based on density to cluster, so that the distance between clusters is greater than the distance threshold δ, and then select the cluster with the highest score, so that the number of words is greater than β×Len, wherein β is a parameter word threshold, and 0<β<1, to obtain a log template; (3) Treat network log data as a rank-3 tensor, i.e. log template, event and time window, and extract the template group and host relationship appearing simultaneously in the log data, as follows: (3.1) Decompose the log data of each time window layer by layer, as follows: (3.1.1) Defining a log template matrix V of size I x L, where if template i belongs to template group l, then the template component v il > 0 of the log template matrix V, and if template i does not belong to template group l, then v il = 0. (3.1.2) Represent an event tensor with a rank-3 tensor Z, each H x L matrix is an event slice, in each event slice, if the template group l on host h belongs to the kth event, then the event component z lkh > 0; otherwise, z lkh = 0; (3.1.3) Let W represent the time window weight matrix, the weight component of the kth event occurring in the jth time window is denoted as w kj , complete the layer-by-layer decomposition of the log tensor, get the log template, event and time window weight component; (3.2) Express the decomposition problem as a minimization problem defined by KL divergence and loss function, and iteratively calculate the components of the log tensor according to the multiplication update rule to extract the events therein; (4) Represent each internal node in the events corresponding to the current log as a flow creation event, and represent the child nodes of the internal node as events of process interacting with system resources, to form an event traceability graph through event connection; (5) Extract event paths and store them by taking a deep traversal method according to the event traceability graph; (6) Model the causal relationship of events by using the extracted event paths, and construct a causal relationship matrix: (6.1) Define the following four-dimensional relationships: Dimension 1: timestamp difference between parent event and child event; Dimension 2: process number of parent event, parent process number of parent event, host number of parent event and process name of parent process; Dimension 3: IP and port of parent event; Dimension 4: type of parent event; (6.2) Extract the four-dimensional relationships defined in step (6.1) for each event path extracted in step (5), to construct the corresponding causal relationship matrix; each event of the matrix corresponds to a one-dimensional row vector of causal relationship, and the size of the matrix is L×8, wherein L represents the path length; (7) Model the neighborhood relationship of events, and construct a neighborhood relationship matrix: (7.1) Use the first N columns of the neighborhood matrix to represent the past and present deviation, N is the number of event types, get the deviation vector D = {D1, D2,..., DN} of the event N}; (7.2) Using Poisson-Gamma mixture distribution to predict the future neighborhood relations of events, the probabilities of each type of event occurring after a unit of time are calculated and stored using a vector P, P = {P1, P2,..., Pn} where n is the number of event types. N} (7.3) Connect vectors D and P to form a field matrix, wherein each event of the matrix corresponds to a one-dimensional row vector of neighborhood relationship, and the size of the matrix is L×16; (8) Model the stage relationship of events, and construct a stage relationship matrix: (8.1) Take four types of variables as input variables, including user profile, events occurred by users on nodes, attack stages where users are located, and attack nodes where users are located, define four factor functions as follows, and capture the relationship between these input variables: First: attack phase s of the multi-step sustainability attack at time t t With the relationship f between the user and the current information of the event e t e (e t ,s t );If event e t occurs, it is considered that the multi-step sustainability attack currently exists in s t This stage;​ The second type: the attack phase s of a multi-step sustainable attack at time t. t With the current event e t Attack phase s at time t-1 t-1 Attack event e t-1 The relationship between the three s (e t-1 ,s t-1 ,e t ,s t If event e t An attack event occurred, and at time t-1, an attack event e occurred. t-1 And time t-1 is located at s t-1 The phase, however, suggests that multi-step sustainability attacks currently exist in s t This stage; The third type: the attack phase s of a multi-step sustainable attack at time t. t With user attribute U, the currently occurring event e t The relationship between the two f u (U,e t ,s t If event e t If an attack occurs and the user has attribute U, then a multi-step persistence attack is considered to currently exist in s. t This stage; Fourth: attack phase s of the multi-step sustainability attack at time t t The attack node Node where the user is, the event e currently occurring t The relationship f between the two n (Node, e t , s t ); if event e t occurs and the user is in attack node Node, then it is considered that the multi-step sustainability attack currently exists in s t this phase; (8.2) Combine all factor functions into a set F = {F e ,F s ,F u ,F n}, where F e , F s , F u , F n represent the sets of factor functions f e (e t ,s t ), f s (e t-1 ,s t-1 ,e t ,s t ), f u (U,e t ,s t ) and f n (Node,e t ,s t ) in the factor graph respectively; let the input of each factor function be X = {e t , U, Node} and Y = {S t}, output 1 when X and Y satisfy the association relationship, otherwise output 0; calculate the conditional probability density P(Y|X); (8.3) Construct a matrix with a size of Lx7 using conditional probability, each event of the matrix corresponds to a one-dimensional length of 7 row vector of neighborhood relationship, representing seven attack stages of multi-step sustainable attack; Set the value of the column corresponding to the stage where each row is most likely to be as P, and the values of the remaining columns as 0 to obtain the stage relationship matrix; (9) For each event path, input the corresponding causal relationship matrix, neighborhood relationship matrix and stage relationship matrix into three layers of long short-term memory neural network LSTM and one layer of Bayesian neural network BNN, and train the classifier using ELBO loss; (10) Use the trained classifier to predict the event path, generate the prediction result of the category, and indicate that the event is benign or multi-step sustainable attack, and generate the uncertainty score representing the prediction standard deviation to realize multi-step sustainable attack detection.

2. The method of claim 1, wherein: The multiplication update rule in step (3.2) is as follows: wherein, The component values corresponding to the component iterations of the old log template, the event and the time window weight respectively; the template words and the parameter words are processed by using the hierarchical decomposition of the log, the decomposed variables are substituted into the updating rule formula for iterative calculation, and the event corresponding to the current log is obtained.

3. The method of claim 1, wherein: In step (5), the event path is extracted and stored in a depth-first manner according to the event trace graph. Specifically, a stack is first initialized to store the event path for depth-first traversal. Then, a leaf node is selected for depth-first traversal. The child nodes are accessed downward, and the accessed child nodes are stacked until the root node. The nodes in the stack at this time constitute an event path. The stack is then popped back to re-traverse the unvisited child nodes, and the event path recording process is repeated until all nodes in the graph are traversed, and all event paths are stored.

4. The method of claim 1, wherein: In step (7), the neighborhood relationship of the event is modeled to represent the past and future properties of the event, where the past refers to the event properties before the event, and the future refers to the probability of future events predicted by the distribution of the event.

5. The method of claim 1, wherein: The deviation vector D = {D1, D2,..., D N} of the event in step (7.1) is obtained, assuming that there are four columns of processes, files, flows, and command shells, and the difference is stored using the deviation vector D = {D process , D file , D flow , D shell}, and the calculation method is as follows: Where timediff is the time elapsed from the initial event to the current event, and Avg(type) is the average probability of type events occurring, calculated as follows: Where sumTime represents the time interval from the initial event to the current new event, and sumNum(type) represents the total number of type events occurring within the sumTime time range.

6. The method of claim 1, wherein: In step (7.2), the probability of each type of event occurring after a unit of time is calculated, i.e., the probability of the Poisson-Gamma mixed distribution is calculated as follows: The probability density function of the Gamma distribution is as follows: Where k and θ are the shape parameters of the Gamma distribution, Γ(k) represents the Gamma function, and λ represents the average number of events per unit time; The probability mass function of the Poisson distribution is as follows: Where k represents the specific number of events; The probability mass function of the Poisson-Gamma mixed distribution is as follows:

7. The method of claim 1, wherein: In step (8.2), the conditional probability density P(Y|X) is calculated as follows: The joint probability density P(E t ,S t ,U,N) is obtained according to the following formula: The conditional probability density P(Y|X) is calculated as follows: Where Z is the normalization factor, and Π represents the multiplication symbol.

Images