Intrusion detection system and detection method

Abstract

The invention provides an intrusion detection system, which is used for analyzing log data to realize intrusion detection, and comprises a composition module used for receiving log data of a preset time window and constructing a window traceability graph based on the log data; the behavior coding module comprises a scoring device and a KAS constructor, and the scoring device is used for scoring the processes in the window traceability graph to obtain the deviation-based abnormal score of each process; the KAS constructor is used for selecting a plurality of candidate processes according to the deviation-based exception score of each process and a preset screening rule, and extracting a key exception set corresponding to each candidate process; the global memory module is used for storing all the candidate processes transmitted by the behavior coding module and the key exception sets corresponding to the candidate processes; the detection module is used for screening abnormal processes from all the candidate processes stored in the global memory module in a fixed detection period according to a preset detection rule; and the alarm module is used for outputting an abnormal behavior graph alarm.

Description

Technical Field

[0001] The present invention relates to the field of information security, in particular to an intrusion detection technology in the field of information security, and more particularly to an intrusion detection system and a detection method. Background Art

[0002] Currently, enterprises and public institutions are facing increasing information security threats, and the attack methods used by attackers are becoming more complex and varied. In order to effectively meet this challenge, researchers use the provenance graph constructed by kernel-level system logs to model the intricate associations between system entities. Among them, system entities (such as processes, files, networks, etc.) are modeled as nodes, and the interactions between them are modeled as edges. The provenance graph-based intrusion detection system (PIDS) can extract contextual information from the provenance graph, and then detect and reveal clues of threats.

[0003] Although it is possible to detect and discover the traces of threats by constructing a provenance graph, attacks such as advanced persistent threats (APTs) often use unknown technologies (e.g., zero-day attacks), which makes it difficult for traditional detection methods to effectively deal with them. To deal with this situation, researchers have proposed anomaly-based detection systems that extract behavioral information from provenance graphs at different granularities and identify behaviors that are different from history as potential threats. Early graph-level detection systems used sketches to abstract and determine the behavior of machines throughout the time slice. In order to further improve the anomaly detection system, some researchers have proposed a node-level detection scheme that can determine potential threats by determining the abnormality of each node. Other researchers have proposed setting RSG (Rooted Subgraph) to represent local structures and learning machine behavior together with graph-level encoding to determine potential threats. Other researchers have proposed that edge-level anomalies can be determined through a temporal model, and then graph-level anomaly scenarios can be reconstructed based on edge-level information.

[0004] Although the anomaly detection system proposed by the prior art can detect potential threats, it still has two defects. On the one hand, it cannot meet the requirements of simplicity and completeness at the same time, that is, it cannot meet the requirements of high precision and high recall at the node level at the same time; simplicity means that the detection output contains almost only threat-related nodes and edges, which can be measured by node-level precision; completeness means that the detection output contains almost all threat-related nodes and edges, which can be measured by node-level recall. On the other hand, it cannot meet the requirements of intuitiveness, that is, it is difficult to fully retain the important related information of the attack scenario in the detection output. Summary of the invention

[0005] Therefore, the purpose of the present invention is to overcome the defects of the prior art and provide an intrusion detection system and an intrusion detection method.

[0006] The objectives of the present invention are achieved through the following technical solutions.

[0007] According to a first aspect of the present invention, an intrusion detection system is provided for analyzing log data to implement intrusion detection, the system comprising: a graphing module for receiving log data of a preset time window and constructing a window traceability graph based thereon; wherein the window traceability graph comprises a plurality of nodes and a plurality of edges connecting two nodes, a node represents a process, a file or an IP appearing within a preset time window, and an edge represents an interaction relationship between the two nodes connected thereto; a behavior coding module comprising a scorer and a KAS constructor, the scorer being used to score processes in the window traceability graph to obtain a deviation-based anomaly score for each process; the KAS constructor being used to select a plurality of candidate processes according to a preset screening rule based on the deviation-based anomaly score of each process, and extract a key anomaly set corresponding to each candidate process; wherein the scorer is based on a process and its neighbors as input, a process-based anomaly score, and a process-based anomaly score. The abnormal neural network model is trained with the deviation anomaly score as the output, and the neighbor of the process refers to the object of the process operation in the same time window; the global memory module is used to store all candidate processes and their corresponding key anomaly sets transmitted by the behavior encoding module, and build a global traceability graph according to all stored candidate processes and their corresponding key anomaly sets; the detection module is used to screen abnormal processes from all candidate processes stored in the global memory module with a fixed detection cycle according to preset detection rules; the alarm module is used to receive the abnormal process transmitted by the detection module, and read the global traceability graph and the key anomaly sets corresponding to all abnormal processes from the global memory module according to all received abnormal processes, and reconstruct the abnormal traceability graph according to the mutual relationship of all abnormal processes in the global traceability graph, and process the abnormal traceability graph according to the preset recognition rules to output an abnormal behavior graph alarm.

[0008] In some embodiments of the present invention, the scorer is trained in the following manner: obtaining log data of multiple preset time windows to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors in the same time window as the process; constructing an initial model, wherein the model includes a feature extractor, a memory encoder, a decoder, and an outlier calculator; wherein the feature extractor is used to extract the behavioral features of the process; the memory encoder is used to extract the behavioral features of all neighbors corresponding to the process; the decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature; the outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process; using the training sample as input and the deviation-based anomaly score of the process as output, updating the parameters of the abnormal neural network model by minimizing the preset training target until the model converges, wherein: the preset training target is:

[0009] RE=MSELoss(I s ′,I s )

[0010] Among them, RE represents the reconstruction error, MSELoss(·) represents the mean square error loss function, and I s ′ represents the reconstruction feature, I s Represents the behavioral characteristics of a process.

[0011] In some embodiments of the present invention, the preset screening rule is: sorting the deviation-based anomaly score of each process in the window traceability graph in order from large to small, and selecting multiple processes that are greater than or equal to the first threshold and ranked first as candidate processes based on a preset first threshold.

[0012] In some embodiments of the present invention, the KAS constructor is configured to extract the key exception set corresponding to each candidate process in the following manner: obtain a neighbor set corresponding to the process based on a window traceability graph constructed based on the time window where the process is located, wherein the neighbor set includes processes, files or IPs corresponding to other nodes that have a connection relationship with the node corresponding to the process; extract files and IPs from the neighbor set, and use a preset whitelist to filter and extract files and IPs in the processed neighbor set that are consistent with the elements in the whitelist, to obtain the key exception set corresponding to the process.

[0013] In some embodiments of the present invention, the preset detection rule is: use the comprehensive scorer configured on the detection module to score each candidate process stored in the global memory module to obtain a comprehensive abnormality value of each candidate process; sort the comprehensive abnormality value of each candidate process in order from large to small, and select multiple candidate processes that are greater than or equal to the second threshold and ranked first as abnormal processes based on a preset second threshold.

[0014] In some embodiments of the present invention, the comprehensive scorer is configured to score each candidate process in the following manner: the rarity score of the candidate process corresponding to the key anomaly set is calculated in the following manner:

[0015]

[0016] Among them, KS κ represents the rarity score of the key anomaly set κ; κ represents the key anomaly set; min(·) represents the minimization function; ln(·) represents the logarithmic function; |W| represents the number of received time windows; represents the number of time windows in which the key anomaly set κ exists and is hit; represents the number of times the key anomaly set κ is hit; It represents the maximum number of times each key anomaly set is hit among all key anomaly sets;

[0017] The rarity score of a candidate process hitting the critical anomaly set is calculated as follows:

[0018]

[0019] Among them, PS p⊥κ represents the rarity score of candidate process p hitting the key anomaly set κ, p⊥κ represents candidate process p hitting the key anomaly set κ; e represents the element in the key anomaly set κ; N(p) represents the neighbor set of candidate process p; represents the number of times element e in the key anomaly set κ is hit;

[0020] The composite outlier value of a candidate process is calculated as follows:

[0021]

[0022] in:

[0023] AS p⊥κ =RE p ×KS κ ×PS p⊥κ

[0024] Among them, AS p represents the comprehensive abnormal value of the candidate process p; Kp represents the set of all key exception sets hit by the candidate process p; AS p⊥κ represents the abnormal value of candidate process p hitting the critical abnormal set κ, HE p represents the anomaly score of the candidate process p based on its deviation.

[0025] In some embodiments of the present invention, the preset identification rule is: dividing the abnormal traceability graph into multiple non-overlapping connected graphs, and calculating the abnormal value of each connected graph according to the preset calculation rule, wherein the preset calculation rule is:

[0026]

[0027] Among them, AS g represents the outlier value of the connected graph g, K g Represents the number of key anomaly sets contained in the connected graph g; based on the outliers of each connected graph, the Grubbs test algorithm is executed to identify the connected graphs with outliers and use them as abnormal behavior graphs for alarm.

[0028] According to a second aspect of the present invention, there is provided an intrusion detection method, the method comprising: step S1, obtaining log data of a historical time window of a device to be detected; step S2, processing the log data of the historical time window of the device to be detected using the intrusion detection system as described in the first aspect of the present invention.

[0029] Compared with the prior art, the advantages of the present invention are: (1) a scorer is set to evaluate the abnormal situation of process nodes in the window traceability graph based on deviation, and candidate processes are selected based on the evaluation situation of each process node, thereby distinguishing normal behavior nodes from abnormal behavior nodes, which is conducive to reducing node redundancy; (2) a KAS constructor is set to construct a key abnormality set for each candidate process, so as to judge the abnormal information of the candidate process according to statistics such as the number of hits of the key abnormality set recorded during runtime, and then find the nodes that are truly related to the attack, which is conducive to reducing node redundancy; (3) an alarm module is set to selectively reconstruct edges based on the abnormal information of the process, and divide and report the abnormal behavior graph through connectivity, so that closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present information related to threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram of the structure of an intrusion detection system according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of the structure of an initial model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] As mentioned in the background technology section, although the anomaly detection system proposed by the prior art can detect potential threats, it still has two defects. On the one hand, it is impossible to simultaneously meet simplicity and completeness, that is, it is impossible to simultaneously meet high precision and high recall rate at the node level; where simplicity means that the detection output contains almost only threat-related nodes and edges, which can be measured by node-level precision; completeness means that the detection output contains almost all threat-related nodes and edges, which can be measured by node-level recall rate. On the other hand, it is impossible to meet intuitiveness, that is, it is difficult to fully retain important related information of the attack scenario in the detection output.

[0035] In order to solve the above problems, the inventors studied the existing anomaly-based traceability graph detection system and found that. The reason why it is impossible to meet both simplicity and completeness is that the existing graph-level anomaly detection system cannot distinguish between different nodes and events in the detected anomaly time slices. This time slice graph contains up to 50,000 nodes. Although the detection is performed in a more fine-grained manner when performing anomaly detection, the scale of nodes reported in the detection output is still large, and can even exceed 10,000 nodes, and there may be only dozens of nodes that are truly related to the attack. The reason why it cannot meet intuitiveness is that the detection output of some existing anomaly detection systems is structured isolated points, which will lose the correlation information between nodes; although the output of some existing anomaly detection systems is a graph, these anomaly detection systems will divide the interrelated nodes and edges in the same threat scenario into different output graphs, and still cannot clearly and comprehensively present information related to the threat.

[0036] Based on the analysis of the prior art, the inventors proposed a new detection system, in which a scorer is set to evaluate the abnormal situation of the process nodes in the window traceability graph based on deviation, and select candidate processes based on the evaluation of each process node. Furthermore, a KAS constructor is also set to construct a key anomaly set for each candidate process, so as to judge the abnormal information of the candidate process based on statistics such as the number of hits of the key anomaly set recorded at runtime, and then find the nodes that are truly related to the attack, reducing node redundancy. Furthermore, an alarm module is set in the detection system, so that the alarm module selectively reconstructs the edges through the abnormal information of the process, and divides and reports the abnormal behavior graph through connectivity, so that closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present information related to the threat.

[0037] In summary, if Figure 1 As shown, the present invention provides an intrusion detection system for analyzing log data to achieve intrusion detection, the system comprising: a graphing module, for receiving log data of a preset time window and constructing a window traceability graph based thereon; wherein the window traceability graph comprises a plurality of nodes and a plurality of edges connecting two nodes, a node represents a process, a file or an IP appearing within a preset time window, and an edge represents an interaction relationship between the two nodes connected thereto; a behavior coding module, comprising a scorer and a KAS constructor, the scorer being used to score the processes in the window traceability graph to obtain a deviation-based anomaly score for each process; the KAS constructor being used to select a plurality of candidate processes according to the deviation-based anomaly score of each process according to a preset screening rule, and extract a key anomaly set corresponding to each candidate process; wherein the scorer is based on a process and its neighbors as input, a deviation-based anomaly score of the process, and a ... the process according to a preset screening rule, and extract a key anomaly set corresponding to each candidate process; wherein the scorer is based on a process and its neighbors as input, a deviation-based anomaly score of the process, and a deviation-based anomaly score of the process. The abnormal neural network model obtained by training is output with the abnormal score of the process, and the neighbor of the process refers to the object of the process operation in the same time window; the global memory module is used to store all candidate processes and their corresponding key abnormal sets transmitted by the behavior encoding module, and to build a global traceability graph according to all stored candidate processes and their corresponding key abnormal sets; the detection module is used to screen abnormal processes from all candidate processes stored in the global memory module with a fixed detection cycle according to preset detection rules; the alarm module is used to receive the abnormal processes transmitted by the detection module, and read the global traceability graph and the key abnormal sets corresponding to all abnormal processes from the global memory module according to all received abnormal processes, and reconstruct the abnormal traceability graph according to the mutual relationship of all abnormal processes in the global traceability graph, and process the abnormal traceability graph according to the preset recognition rules to output abnormal behavior graph alarm.

[0038] In order to better understand the present invention, each component module in the intrusion detection system is described in detail below in conjunction with specific embodiments.

[0039] 1. Composition module

[0040] The mapping module is used to receive the log data of a preset time window and construct a window traceability graph based on it. Among them, the window traceability graph includes multiple nodes and multiple edges connecting two nodes. The node represents the process, file or IP that appears in the preset time window, and the edge represents the interaction relationship between the two nodes it is connected to. It should be noted that the time window can be set to 15 minutes, or it can be set according to actual needs, and the present invention does not make special restrictions.

[0041] 2. Behavior Coding Module

[0042] The behavior encoding module is used to receive and process the window traceability graph from the composition module, and includes a scorer and a KAS constructor, wherein the scorer is used to score the processes in the window traceability graph to obtain the deviation-based anomaly score of each process; the KAS constructor is used to select multiple candidate processes according to the deviation-based anomaly score of each process according to the preset screening rules, and extract the key anomaly set corresponding to each candidate process. The scorer and KAS constructor are introduced below.

[0043] The scorer is used to score the processes in the window traceability graph to obtain a deviation-based anomaly score for each process, wherein the scorer is an anomaly neural network model trained with the process and its neighbors as input and the deviation-based anomaly score of the process as output.

[0044] According to one embodiment of the present invention, the scorer is trained in the following manner: first, log data of multiple preset time windows are obtained to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors in the same time window as the process (the neighbors are objects operated by the process, such as written files, forked child processes, etc.); then, the scorer is constructed as follows: Figure 2 The initial model shown in the figure includes a feature extractor, a memory encoder, a decoder and an outlier calculator; wherein the feature extractor is used to extract the behavioral features of the process; the memory encoder is used to extract the behavioral features of all neighbors corresponding to the process; the decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature; the outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process; finally, with the training sample as input and the deviation-based anomaly score of the process as output, the parameters of the abnormal neural network model are updated by minimizing the preset training target until the model converges, wherein the preset training target is: RE=MSELoss(I s ′,I s), where RE represents the reconstruction error, MSELoss(·) represents the mean square error loss function, and I s ′ represents the reconstruction feature, I s It indicates the behavioral characteristics of the process. It should be noted that both the feature extractor and the decoder are multi-layer perceptrons, and the memory encoder is composed of a memory module, a linear layer, and a pooling layer.

[0045] The KAS constructor is used to select multiple candidate processes according to the deviation-based anomaly score of each process according to a preset screening rule, and extract a key anomaly set corresponding to each candidate process.

[0046] According to an embodiment of the present invention, the preset screening rule is: sort the deviation-based anomaly score of each process in the window traceability graph in descending order, and select multiple processes that are greater than or equal to the first threshold and ranked first as candidate processes based on a preset first threshold. It should be noted that the first threshold can be determined according to actual needs, and the present invention does not impose specific restrictions. For example, the first threshold can be set to α (0<α<1).

[0047] According to one embodiment of the present invention, the KAS constructor is configured to extract the key anomaly set corresponding to each candidate process in the following manner: obtain the neighbor set corresponding to the process based on the window traceability graph constructed in the time window where the process is located, wherein the neighbor set includes processes, files or IPs corresponding to other nodes that have a connection relationship with the node corresponding to the process; extract the files and IPs in the neighbor set, and use the preset whitelist to filter and extract the files and IPs in the neighbor set that are consistent with the elements in the whitelist, and obtain the key anomaly set corresponding to the process. Among them, the preset whitelist is extracted from the training data constructed when training the scorer. The extraction of the whitelist needs to meet two principles. The first is that the file or IP appears in more than 10% of the historical windows (the time window appearing in the training data); the second is that the file or IP is accessed by a large number of processes, and a large number refers to the value determined by multiplying the total number of processes by 0.001. The file or IP is accessed by processes exceeding this value and is regarded as being accessed in large quantities. It should be noted that when extracting the key anomaly set corresponding to the candidate process, the file name involved in the neighbor set needs to be transformed into a coarsened file element by regular expression matching. It should also be noted that if the key exception set corresponding to the extracted candidate process is an empty set, it will be discarded.

[0048] 3. Global Memory Module

[0049] The global memory module is used to store all candidate processes and their corresponding key exception sets transmitted by the behavior coding module, and to construct a global traceability graph based on all stored candidate processes and their corresponding key exception sets. It should be noted that the key exception set corresponding to the candidate process will be merged with other similar key exception sets in the global memory module. At this time, it can be regarded as the candidate process hitting other key exception sets, where similarity refers to the intersection ratio between the key exception set corresponding to the candidate process and other key exception sets is greater than or equal to the preset value, which can be set to 0.8 or determined according to actual needs. It should also be noted that since the composition module will continuously process the log data of the preset time window, the behavior coding module will also continue to receive and process the window traceability graph from the composition module, and transmit the candidate process and its corresponding key exception set obtained by the behavior coding module to the global memory module. In this case, the candidate processes and key exception sets corresponding to the processes stored in the global memory module will increase, and the global traceability graph constructed by it will also be continuously updated.

[0050] 4. Detection Module

[0051] The detection module is used to screen abnormal processes from all candidate processes stored in the global memory module according to preset detection rules in a fixed detection cycle. The fixed detection cycle can be 24 hours, that is, when the data processed by the composition module and the behavior encoding module reaches the detection cycle, the detection module obtains abnormal processes related to abnormal behaviors from the global memory module.

[0052] According to an embodiment of the present invention, the preset detection rule is: first, use the comprehensive scorer configured on the detection module to score each candidate process stored in the global memory module to obtain the comprehensive abnormality value of each candidate process; sort the comprehensive abnormality value of each candidate process in order from large to small, and select multiple candidate processes that are greater than or equal to the second threshold and ranked first as abnormal processes based on the preset second threshold. It should be noted that the second threshold can be determined according to actual needs, and the present invention does not make specific restrictions. For example, the second threshold can be set to β (0<β<1).

[0053] According to an embodiment of the present invention, the comprehensive scorer is configured to score each candidate process according to the following steps. Each step is described below.

[0054] First, the rarity score of the key anomaly set corresponding to the candidate process is calculated as follows:

[0055]

[0056] Among them, KS κrepresents the rarity score of the key anomaly set κ; κ represents the key anomaly set; ln(·) represents the logarithmic function; min(·) represents the minimization function; |W| represents the number of received time windows; Indicates the number of time windows in which the critical anomaly set κ exists and is hit. For example, for a specified critical anomaly set κ1, assuming that processes p1 and p2 hit the critical anomaly set κ1 in time window t1, and process p3 hits the critical anomaly set κ1 in time window t2, then the time windows in which the specified critical anomaly set κ1 exists and is hit are t1 and t2. Indicates the number of times the key exception set κ is hit. For example, for a specified key exception set κ1, it is hit by three processes p1, p2, and p3. It represents the maximum number of times each key anomaly set is hit among all key anomaly sets. For example, for key anomaly set κ1, For the key anomaly set κ2, For the key anomaly set κ3, Among them, the key anomaly set κ2 corresponds to The maximum is based on the key anomaly sets κ1, κ2 and κ3 corresponding to

[0057] Then, the rarity score of the candidate process hitting the critical anomaly set is calculated as follows:

[0058]

[0059] Among them, PS p⊥κ represents the rarity score of candidate process p hitting the key anomaly set κ, p⊥κ represents candidate process p hitting the key anomaly set κ (hitting means that the intersection ratio between the key anomaly set corresponding to the process and other key anomaly sets is greater than or equal to the preset value); e represents an element in the key anomaly set κ; N(p) represents the neighbor set of candidate process p; Indicates the number of times element e in the key exception set κ is hit. For example, for a specified key exception set κ1, which is hit by processes p1, p2, and p3, the number of times element e in the key exception set κ1 is hit refers to the number of elements in the key exception sets corresponding to processes p1, p2, and p3 that are consistent with element e.

[0060] Finally, the comprehensive abnormality value of the candidate process is calculated as follows:

[0061]

[0062] AS p⊥κ =RE p ×KS κ×PS p⊥κ

[0063] Among them, AS p represents the comprehensive abnormal value of the candidate process p; K p represents the set of all key exception sets hit by the candidate process p; AS p⊥κ represents the abnormal value of candidate process p hitting the critical abnormal set κ, RE p represents the anomaly score of the candidate process p based on its deviation.

[0064] It should be noted that the abnormal processes obtained by screening based on the above-mentioned embodiment simultaneously meet the requirements of large deviation error and rare behavior, and thus, based on the screened abnormal processes, it is possible to better determine which information is related to threats.

[0065] 5. Alarm module

[0066] The alarm module is used to receive the abnormal process transmitted by the detection module, and read the global traceability graph and the key exception set corresponding to each of all abnormal processes from the global memory module according to all the received abnormal processes, and reconstruct the abnormal traceability graph according to the mutual relationship of all abnormal processes in the global traceability graph, and process the abnormal traceability graph according to the preset recognition rules to output the abnormal behavior graph alarm. It should be noted that when reconstructing the abnormal traceability graph, the direction of the edge can be defined as the direction of the dependency flow, wherein the dependency flow includes the control flow and the data flow, and the control flow exists in the process replicating the process, for example, process p1 replicating (fork) process p2, at this time the direction of the control flow is from process p1 to p2; the data flow exists in the process accessing the file or IP, for example, process p1 reads (read) file f, at this time the data flow direction is from f to p1.

[0067] According to an embodiment of the present invention, the preset identification rule is: dividing the abnormal traceability graph into multiple non-overlapping connected graphs, and calculating the abnormal value of each connected graph according to the preset calculation rule, wherein the preset calculation rule is:

[0068]

[0069] Among them; AS g represents the outlier value of the connected graph g, K g represents the number of key anomaly sets contained in the connected graph g, AS p⊥κ Indicates that the candidate process p hits the abnormal value of the key abnormal set κ, and at the abnormal value of each connected graph, p also represents the first process of the connected graph g hitting the key abnormal set κ; based on the abnormal value of each connected graph, the Grubbs test algorithm is executed to identify the connected graph with outlier abnormal values, and it is used as an abnormal behavior graph alarm, wherein the execution process of the Grubbs test algorithm can be expressed as follows:

[0070] AS alert =Outlier({AS g |g∈G candidate})

[0071] G alert ={g|AS g ∈AS alert})

[0072] Among them, Outlier(·) represents the Grubbs test method, AS alert The connected graph set representing the outliers, G alert It represents a collection of abnormal behavior graphs reported. It should be noted that in addition to the Grubbs test algorithm, other algorithms such as K-nearest neighbor, isolation forest, and local outlier factor can also be used.

[0073] The intrusion detection system described in the above embodiment can be used to perform intrusion detection. Based on this, the present invention also proposes an intrusion detection method, which includes: step S1, obtaining the log data of the historical time window of the device to be detected; step S2, using the intrusion detection system described in the above embodiment to process the log data of the historical time window of the device to be detected. Among them, the specific intrusion detection process involved in step S2 is: the composition module first receives the log data of the preset time window and constructs the window traceability graph; then, the behavior coding module processes the window traceability graph to obtain multiple candidate processes and their corresponding key anomaly sets, and transmits the multiple candidate processes and their corresponding key anomaly sets to the global memory module for storage; the detection module determines whether the detection cycle has been reached. If the detection cycle has been reached, the abnormal process is filtered from the global memory module and output to the alarm module, and the alarm module processes and outputs the abnormal behavior graph alarm. It should also be noted that the device to be detected can be a host or terminal system such as a Linux host, an Apple operating system, a Microsoft operating system, and an Android operating system.

[0074] In order to better understand that the present invention has a better intrusion detection effect than the prior art, the present invention sets up a comparative experiment to evaluate the intrusion detection performance of the prior art and the present invention.

[0075] In the comparison experiment, the CADETS dataset and THEIA dataset in the existing DARPA TC dataset are used as test data to verify the detection performance between the prior art and the intrusion detection system proposed in the present invention. The experimental results shown in Table 1, Table 2 and Table 3 are obtained.

[0076] Among them, Table 1 shows the accuracy and recall rate of the intrusion detection system proposed in the present invention in detecting abnormal processes, abnormal files, and abnormal IPs on different data under different attack dates. It can be seen from Table 1 that the intrusion detection system proposed in the present invention has a great advantage in recall rate and accuracy at the node level.

[0077] Table 2 shows the performance comparison between the intrusion detection system proposed in the present invention and the prior art on the same data segment. As can be seen from Table 2, no matter which data set is used, the intrusion detection system proposed in the present invention is superior to the prior art (ThreaTrace) in terms of accuracy and recall rate in detecting abnormal processes, abnormal files, and abnormal IPs.

[0078] Table 3 shows the comparison of the number of abnormal behavior graphs output by the intrusion detection system proposed in the present invention and the prior art on the same data segment. In Table 3, "#Abnormal Graphs" refers to the number of abnormal behavior graphs output, and "Other Graphs" refers to the number of normal behavior graphs output. It can be seen from Table 3 that, compared with the prior art, the intrusion detection system proposed in the present invention outputs fewer normal graphs and can use fewer abnormal behavior graphs to display information related to the threat scenario.

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] Table 3

[0084]

[0085] The beneficial effects of the present invention are: (1) a scorer is set to evaluate the abnormal situation of process nodes in the window traceability graph based on deviation, and a candidate process is selected based on the evaluation situation of each process node, thereby distinguishing normal behavior nodes from abnormal behavior nodes, which is conducive to reducing node redundancy; (2) a KAS constructor is set to construct a key abnormality set for each candidate process, so as to judge the abnormal information of the candidate process according to statistics such as the number of hits of the key abnormality set recorded at runtime, thereby finding the nodes that are truly related to the attack, which is conducive to reducing node redundancy; (3) an alarm module is set to selectively reconstruct edges based on the abnormal information of the process, and divide and report the abnormal behavior graph through connectivity, so that closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present information related to the threat.

[0086] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0087] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0088] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0089] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An intrusion detection system for analyzing log data to implement intrusion detection, characterized in that: The system comprises: A graph construction module is used to receive log data of a preset time window and construct a window traceability graph based on the log data; wherein the window traceability graph includes multiple nodes and multiple edges connecting two nodes, where a node represents a process, file or IP that appears in the preset time window, and an edge represents the interaction relationship between the two nodes connected thereto; The behavior coding module includes a scorer and a KAS constructor. The scorer is used to score the processes in the window traceability graph to obtain the deviation-based anomaly score of each process. The KAS constructor is used to select multiple candidate processes according to the deviation-based anomaly score of each process according to the preset screening rules, and extract the key anomaly set corresponding to each candidate process. The scorer is an abnormal neural network model trained with the process and its neighbors as input and the deviation-based anomaly score of the process as output. The neighbors of the process refer to the objects operated by the process in the same time window. A global memory module is used to store all candidate processes and their corresponding key anomaly sets transmitted by the behavior encoding module, and to construct a global traceability graph based on all stored candidate processes and their corresponding key anomaly sets; A detection module, used to screen abnormal processes from all candidate processes stored in the global memory module according to preset detection rules and at a fixed detection cycle; An alarm module is used to receive the abnormal process transmitted by the detection module, and read the global traceability graph and the key exception sets corresponding to all abnormal processes from the global memory module according to all the received abnormal processes, and reconstruct the abnormal traceability graph according to the mutual relationship of all abnormal processes in the global traceability graph, and process the abnormal traceability graph according to the preset recognition rules to output an abnormal behavior graph alarm.

2. The system according to claim 1, characterized in that The scorer is trained in the following way: Acquire log data of multiple preset time windows to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors in the same time window as the process; Construct an initial model, which includes a feature extractor, a memory encoder, a decoder, and an outlier calculator; wherein the feature extractor is used to extract the behavioral features of the process; the memory encoder is used to extract the behavioral features of all neighbors corresponding to the process; the decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature; the outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process; With the training sample as input and the deviation-based anomaly score of the process as output, the parameters of the anomaly neural network model are updated by minimizing the preset training target until the model converges, where the preset training target is: RE=MSELoss(I s ′,I s ) Among them, RE represents the reconstruction error, MSELoss(·) represents the mean square error loss function, and I s ′ represents the reconstruction feature, I s Represents the behavioral characteristics of a process.

3. The system according to claim 2, characterized in that The preset screening rules are: The deviation-based anomaly score of each process in the window traceability graph is sorted in descending order, and a plurality of processes that are greater than or equal to the first threshold and ranked first are selected as candidate processes based on a preset first threshold.

4. The system according to claim 3, characterized in that The KAS constructor is configured to extract the key exception set corresponding to each candidate process in the following manner: Obtain a neighbor set corresponding to the process based on a window traceability graph constructed based on the time window of the process, wherein the neighbor set includes processes, files or IPs corresponding to other nodes that have a connection relationship with the node corresponding to the process; The files and IPs in the neighbor set are extracted, and the files and IPs in the neighbor set after the extraction process that are consistent with the elements in the whitelist are filtered using a preset whitelist to obtain a key exception set corresponding to the process.

5. The system according to claim 4, characterized in that The preset detection rules are: The comprehensive scorer configured on the detection module is used to score each candidate process stored in the global memory module to obtain the comprehensive abnormal value of each candidate process; The comprehensive abnormal value of each candidate process is sorted in descending order, and a plurality of candidate processes whose values ​​are greater than or equal to the second threshold and are ranked first are selected as abnormal processes based on a preset second threshold.

6. The system according to claim 5, characterized in that The comprehensive scorer is configured to score each candidate process as follows: The rarity score of the key anomaly set corresponding to the candidate process is calculated as follows: Among them, KS κ represents the rarity score of the key anomaly set K; κ represents the key anomaly set; min(·) represents the minimization function; ln(·) represents the logarithmic function; |W| represents the number of received time windows; Indicates the number of time windows in which the key anomaly set K exists and is hit; represents the number of times the key anomaly set κ is hit; It represents the maximum number of times each key anomaly set is hit among all key anomaly sets; The rarity score of a candidate process hitting the critical anomaly set is calculated as follows: Among them, PS p⊥κ represents the rarity score of candidate process p hitting the key anomaly set κ, p⊥κ represents candidate process p hitting the key anomaly set K; e represents an element in the key anomaly set K; N(p) represents the neighbor set of candidate process p; Indicates the number of times element e in the key anomaly set K is hit; The composite outlier value of a candidate process is calculated as follows: in: AS p⊥κ =RE p ×KS κ ×PS p⊥κ Among them, AS p represents the comprehensive abnormal value of the candidate process p; K p represents the set of all key exception sets hit by the candidate process p; AS p⊥κ represents the abnormal value of candidate process p hitting the key abnormal set K, RE p represents the anomaly score of the candidate process p based on its deviation.

7. The system according to claim 6, characterized in that The preset identification rule is: The anomaly tracing graph is divided into multiple non-overlapping connected graphs, and the anomaly value of each connected graph is calculated according to the preset calculation rules, where the preset calculation rules are: Among them, AS g represents the outlier value of the connected graph g, K g represents the number of key anomaly sets contained in the connected graph g; The Grubbs test algorithm is performed based on the outliers of each connectivity graph to identify connectivity graphs with outliers and alarm them as abnormal behavior graphs.

8. An intrusion detection method, characterized in that: The method comprises: Step S1, obtaining log data of a historical time window of a device to be detected; Step S2: Process the log data of the historical time window of the device to be detected using the intrusion detection system as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to claim 8.

10. An electronic device, characterized in that: include: one or more processors, and A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of claim 8 by executing the executable instructions.

Images