An intrusion detection system, detection method

Abstract

The application provides an intrusion detection system for analyzing log data to realize intrusion detection, the system comprising: a composition module for receiving log data of a preset time window and constructing a window trace graph based thereon; a behavior coding module comprising a scorer and a KAS constructor, the scorer being used for scoring processes in the window trace graph to obtain an abnormality score based on deviation for each process; the KAS constructor being used for selecting a plurality of candidate processes according to the abnormality score based on deviation for each process according to a preset screening rule, and extracting a key abnormality set corresponding to each candidate process; a global memory module for storing all candidate processes and the corresponding key abnormality sets transmitted by the behavior coding module; a detection module for screening abnormal processes from all candidate processes stored in the global memory module according to a preset detection rule at a fixed detection period; and an alarm module for outputting an abnormal behavior graph alarm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information security, in particular to intrusion detection technology in the field of information security, and more particularly to an intrusion detection system and a detection method. BACKGROUND

[0002] Currently, enterprises and public institutions are facing increasingly severe information security threats, and attackers are using more and more complex and varied attack methods. In order to effectively respond to this challenge, researchers use provenance graphs constructed by kernel-level system logs to model the complex relationships between system entities. System entities such as processes, files, networks are modeled as nodes, and their interaction relationships are modeled as edges. The provenance graph-based intrusion detection system (PIDS) can extract context information from the provenance graph, and then detect and reveal the traces of threats.

[0003] Although the provenance graph can be constructed to detect and discover the traces of threats, attacks such as advanced persistent threats (APTs) often use unknown technologies (e.g., zero-day attacks), which makes it difficult for traditional detection methods to effectively respond. In order to deal with this situation, researchers have proposed anomaly-based detection systems that extract behavior information from the provenance graph at different granularities and determine behaviors that are different from history as potential threats. Early graph-level detection systems use sketches to abstract and determine the behavior of machines in the entire time slice. To further improve anomaly detection systems, researchers have proposed node-level detection schemes that can determine potential threats by determining the abnormality of each node. Researchers have also proposed that RSG (Rooted Subgraph) can be set to represent local structures and learn machine behavior together with graph-level encoding to determine potential threats. Researchers have also proposed that temporal models can be used to determine edge-level anomalies, and then reconstruct graph-level anomaly scenarios based on edge-level information.

[0004] Although the anomaly detection systems proposed by the prior art can detect potential threats, there are still two defects. On the one hand, it is impossible to satisfy both simplicity and completeness, i.e., it is impossible to satisfy both high precision and high recall rate at the node level; where simplicity refers to the detection output containing almost only threat-related nodes and edges, which can be measured by node-level precision; completeness refers to the detection output containing almost all threat-related nodes and edges, which can be measured by node-level recall rate. On the other hand, it is impossible to satisfy intuitiveness, i.e., it is difficult to completely retain important association information of attack scenarios in the detection output. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the above-mentioned defects of the prior art, and to provide an intrusion detection system and an intrusion detection method.

[0006] The purpose of the present application is achieved by the following technical solutions.

[0007] According to a first aspect of the present application, an intrusion detection system is provided for analyzing log data to achieve intrusion detection, the system comprising: a composition module for receiving log data of a preset time window and constructing a window trace graph based thereon; wherein the window trace graph comprises a plurality of nodes and a plurality of edges connecting two nodes, the nodes representing processes, files or IPs appearing within the preset time window, and the edges representing the interaction relationship between the two nodes connected thereby; a behavior coding module comprising a scorer and a KAS constructor, the scorer being configured to score the processes in the window trace graph to obtain an abnormal score based on deviation for each process; the KAS constructor being configured to select a plurality of candidate processes according to the abnormal score based on deviation of each process according to a preset screening rule, and extract a key abnormal set corresponding to each candidate process; wherein the scorer is an abnormal neural network model trained with processes and their neighbors as input and the abnormal score based on deviation of the processes as output, the neighbors of the processes referring to the objects operated by the processes within the same time window; a global memory module for storing all candidate processes and their corresponding key abnormal sets transmitted by the behavior coding module, and constructing a global trace graph according to all candidate processes and their corresponding key abnormal sets stored; a detection module for screening abnormal processes from all candidate processes stored in the global memory module according to a preset detection rule with a fixed detection period; an alarm module for receiving the abnormal processes transmitted by the detection module, and reading the global trace graph and the key abnormal sets corresponding to all abnormal processes respectively from the global memory module according to the received all abnormal processes, reconstructing an abnormal trace graph according to the mutual relationship of all abnormal processes in the global trace graph, and processing the abnormal trace graph according to a preset identification rule to output an abnormal behavior graph alarm.

[0008] In some embodiments of the present invention, the scorer is trained in the following manner: obtaining log data of multiple preset time windows to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors in the same time window as the process; constructing an initial model, wherein the model includes a feature extractor, a memory encoder, a decoder, and an outlier calculator; wherein the feature extractor is used to extract the behavioral features of the process; the memory encoder is used to extract the behavioral features of all neighbors corresponding to the process; the decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature; the outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process; using the training sample as input and the deviation-based anomaly score of the process as output, updating the parameters of the anomaly neural network model by minimizing the preset training target until the model converges, wherein: the preset training target is:

[0009] RE=MSELoss(I s ′,I s )

[0010] Among them, RE represents the reconstruction error, MSELoss(·) represents the mean square error loss function, and I s ′ represents the reconstructed feature, I s Indicates the behavioral characteristics of a process.

[0011] In some embodiments of the present invention, the preset screening rule is: sorting the deviation-based anomaly score of each process in the window traceability graph in descending order, and selecting multiple processes that are greater than or equal to the first threshold and ranked first as candidate processes based on a preset first threshold.

[0012] In some embodiments of the present invention, the KAS constructor is configured to extract the key exception set corresponding to each candidate process in the following manner: obtain the neighbor set corresponding to the process based on the window traceability graph constructed based on the time window where the process is located, wherein the neighbor set includes processes, files or IPs corresponding to other nodes that have a connection relationship with the node corresponding to the process; extract the files and IPs in the neighbor set, and use a preset whitelist to filter the extracted files and IPs in the processed neighbor set that are consistent with the elements in the whitelist to obtain the key exception set corresponding to the process.

[0013] In some embodiments of the present application, the preset detection rule is that a comprehensive scorer configured on the detection module scores each candidate process stored in the global memory module to obtain a comprehensive anomaly value of each candidate process; the comprehensive anomaly values of each candidate process are sorted in descending order, and a plurality of candidate processes greater than or equal to a preset second threshold and in front of the sorting are selected as abnormal processes.

[0014] In some embodiments of the present application, the comprehensive scorer is configured to score each candidate process in the following manner: the rarity score of the candidate process corresponding to the key anomaly set is calculated in the following manner:

[0015]

[0016] wherein, KS κ represents the rarity score of the key anomaly set κ; κ represents the key anomaly set; min(·) represents the minimum function; ln(·) represents the logarithmic function; |W| represents the number of received time windows; represents the number of time windows in which the key anomaly set κ is hit; represents the number of times the key anomaly set κ is hit; represents the maximum value of the number of times each key anomaly set is hit in all key anomaly sets;

[0017] The rarity score of the candidate process hitting the key anomaly set is calculated in the following manner:

[0018]

[0019] wherein, PS p⊥κ represents the rarity score of the candidate process p hitting the key anomaly set κ; p represents the candidate process p hitting the key anomaly set κ; e represents an element in the key anomaly set κ; N(p) represents the neighbor set of the candidate process p; represents the number of times the element e in the key anomaly set κ is hit;

[0020] The comprehensive anomaly value of the candidate process is calculated in the following manner:

[0021]

[0022] wherein:

[0023] AS p⊥κ = RE p × KS κ × PS p⊥κ

[0024] wherein, AS p represents the comprehensive anomaly value of the candidate process p; Kp denotes a set of all key anomaly set groups that candidate process p hits; AS p⊥κ denotes the abnormal value of candidate process p hitting key anomaly set K; HE p denotes the abnormal score of candidate process p based on deviation.

[0025] In some embodiments of the present application, the preset identification rule is to divide the abnormality trace graph into a plurality of non-overlapping connected graphs, and calculate the abnormal value of each connected graph according to the preset calculation rule, wherein the preset calculation rule is:

[0026]

[0027] denotes the abnormal value of connected graph g; K g denotes the abnormal value of connected graph g; K g denotes the number of key anomaly sets contained in connected graph g; the Grubbs test algorithm is performed based on the abnormal value of each connected graph to identify the connected graph with an outlier abnormal value, and the connected graph is reported as an abnormal behavior graph.

[0028] According to a second aspect of the present application, an intrusion detection method is provided, which comprises the following steps: S1, obtaining log data of a historical time window of a to-be-detected device; S2, processing the log data of the historical time window of the to-be-detected device by using the intrusion detection system according to the first aspect of the present application.

[0029] Compared with the prior art, the present application has the following advantages: (1) a scorer is arranged to evaluate the abnormality of process nodes in the window trace graph based on deviation, and a candidate process is selected based on the evaluation of each process node, so as to distinguish normal behavior nodes and abnormal behavior nodes, which is beneficial to reduce node redundancy; (2) a KAS constructor is arranged to construct a key anomaly set for each candidate process, so as to judge the abnormal information of the candidate process according to the hit frequency and other statistical quantities of the key anomaly set recorded in the runtime, and then find the nodes that are truly related to attacks, which is beneficial to reduce node redundancy; (3) an alarm module is arranged to selectively reconstruct edges through the abnormal information of the process, and divide and report abnormal behavior graphs through connectivity, so that closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present information related to threats. BRIEF DESCRIPTION OF DRAWINGS

[0030] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0031] Figure 1 FIG. 1 is a structural schematic diagram of an intrusion detection system according to an embodiment of the present application;

[0032] Figure 2 FIG. 2 is a structural schematic diagram of an initial model according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] For the purposes of the present application, the technical solutions and advantages will be clearer. The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are intended to explain the present application and are not intended to limit the present application.

[0034] As mentioned in the background section, although the existing abnormality detection system can detect potential threats, there are still two defects. On the one hand, it is difficult to meet the simplicity and completeness at the same time, that is, it is difficult to meet the high precision and high recall rate at the node level at the same time; wherein the simplicity refers to the detection output containing almost only threat-related nodes and edges, which can be measured by the precision at the node level; the completeness refers to the detection output containing almost all threat-related nodes and edges, which can be measured by the recall rate at the node level. On the other hand, it is difficult to meet the intuitiveness, that is, it is difficult to completely retain the important association information of the attack scene in the detection output.

[0035] In order to solve the above problems, the inventors have found that the existing abnormality-based traceability graph detection system cannot distinguish the different nodes and events in the detected abnormal time slice. Such a time slice graph contains up to 50,000 nodes, and although the detection is performed in a more fine-grained manner when performing abnormality detection, the number of nodes reported in the detection output is still large, even more than 10,000 nodes, and the nodes that are really related to the attack may only be a few dozen. The reason why it is difficult to meet the intuitiveness is that the detection output of some existing abnormality detection systems is a structured isolated point, which will lose the association information between nodes; although the output of some existing abnormality detection systems is a graph, these abnormality detection systems will divide the nodes and edges associated with each other in the same threat scene into different output graphs, and still cannot clearly and comprehensively present the information related to the threat.

[0036] Based on the analysis of the prior art, the inventors propose a new detection system in which a scorer is arranged to evaluate the abnormality of the process nodes in the window trace graph based on deviation, and to select candidate processes based on the evaluation of each process node. Further, a KAS constructor is also arranged to construct a key anomaly set for each candidate process, so as to judge the abnormal information of the candidate process according to the hit frequency and other statistical quantities of the key anomaly set recorded in the runtime, and to find the nodes that are truly related to the attack, thereby reducing the node redundancy. Further, an alarm module is arranged in the detection system, so that the alarm module selectively reconstructs the edges through the abnormal information of the processes, and divides and reports the abnormal behavior graph through the connectivity, so that the closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present the information related to the threat.

[0037] In summary, as shown in Figure 1 In summary, as shown in

[0038] In order to better understand the present application, each component module in the intrusion detection system will be described in detail below in combination with specific embodiments.

[0039] 1. Composition Module

[0040] The mapping module is used to receive log data of a preset time window and construct a window traceability graph based on it. Among them, the window traceability graph includes multiple nodes and multiple edges connecting two nodes. The nodes represent the processes, files or IPs that appear in the preset time window, and the edges represent the interactive relationship between the two nodes they are connected to. It should be noted that the time window can be set to 15 minutes, or it can be set according to actual needs, and the present invention does not impose any special restrictions.

[0041] 2. Behavior Coding Module

[0042] The behavior encoding module receives and processes the window traceability graph from the graph composition module. It includes a scorer and a KAS constructor. The scorer scores the processes in the window traceability graph to obtain a deviation-based anomaly score for each process. The KAS constructor selects multiple candidate processes based on each process's deviation-based anomaly score according to preset screening rules and extracts the key anomaly set corresponding to each candidate process. The scorer and KAS constructor are described below.

[0043] The scorer is used to score the processes in the window traceability graph to obtain a deviation-based anomaly score for each process. The scorer is an anomaly neural network model trained with the process and its neighbors as input and the deviation-based anomaly score of the process as output.

[0044] According to one embodiment of the present invention, the scorer is trained in the following manner: first, log data of multiple preset time windows are obtained to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors of the process in the same time window (the neighbors are objects operated by the process, such as written files, forked child processes, etc.); then, a scorer is constructed as follows: Figure 2 The initial model shown in the figure includes a feature extractor, a memory encoder, a decoder and an outlier calculator; wherein the feature extractor is used to extract the behavioral features of the process; the memory encoder is used to extract the behavioral features of all neighbors corresponding to the process; the decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature; the outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process; finally, with the training sample as input and the deviation-based anomaly score of the process as output, the parameters of the anomaly neural network model are updated by minimizing the preset training target until the model converges, wherein the preset training target is: RE=MSELoss(I s ′,I s), where RE represents a reconstruction error, MSELoss(·) represents a mean square error loss function, I s represents a reconstructed feature, I s represents a behavior feature of the process. It should be noted that the feature extractor and the decoder are both multi-layer perceptrons, and the memory encoder is composed of a memory module, a linear layer and a pooling layer.

[0045] The KAS constructor is configured to select multiple candidate processes according to the deviation-based anomaly scores of each process according to a preset screening rule, and extract a key anomaly set corresponding to each candidate process.

[0046] According to an embodiment of the present application, the preset screening rule is that the deviation-based anomaly scores of each process in the window trace graph are sorted in descending order, and multiple processes greater than or equal to a preset first threshold and in the front of the sorting are selected as candidate processes. It should be noted that the first threshold can be determined according to actual needs, and the present application does not make specific limitations. For example, the first threshold can be set as a (0< a< 1).

[0047] According to an embodiment of the present application, the KAS constructor is configured to extract the key anomaly set corresponding to each candidate process in the following manner: obtaining a neighbor set corresponding to the process based on the window trace graph constructed by the time window in which the process is located, wherein the neighbor set includes processes, files or IPs corresponding to other nodes having a connection relationship with the node corresponding to the process; extracting the files and IPs in the neighbor set, and filtering the files and IPs in the processed neighbor set that are consistent with the elements in the preset whitelist using the preset whitelist, to obtain the key anomaly set corresponding to the process. Wherein, the preset whitelist is extracted from the training data constructed when the training scorer is trained. Two principles need to be met when extracting the whitelist. The first is that the file or IP appears in more than 10% of the historical windows (the time window appears in the training data); the second is that the file or IP is accessed by a large number of processes. A large number refers to a value determined after multiplying the total number of processes by 0.001, and a file or IP accessed by more than this value is considered to be accessed by a large number of processes. It should be noted that when extracting the key anomaly set corresponding to the candidate process, the file name involved in the neighbor set needs to be transformed into a coarse file element through a regular expression matching.

[0048] III. Global memory module

[0049] The global memory module is configured to store all candidate processes and corresponding key abnormal sets transmitted by the behavior coding module, and construct a global trace graph according to all stored candidate processes and corresponding key abnormal sets. It should be noted that the key abnormal set corresponding to the candidate process can be merged with other similar key abnormal sets in the global memory module, and at this time, the candidate process can be considered to hit other key abnormal sets. Similarity refers to that the intersection ratio between the key abnormal set corresponding to the candidate process and other key abnormal sets is greater than or equal to a preset value, which can be set to 0.8, or can be determined according to actual needs. It should be further noted that since the graph construction module will continuously process log data in a preset time window, the behavior coding module will also continuously receive and process window trace graphs from the graph construction module, and transmit candidate processes and corresponding key abnormal sets processed by the behavior coding module to the global memory module. In this case, the candidate processes and corresponding key abnormal sets stored in the global memory module will become more and more, and the global trace graph constructed by the global memory module will also be continuously updated.

[0050] IV. Detection module

[0051] The detection module is configured to screen abnormal processes from all candidate processes stored in the global memory module according to a preset detection rule at a fixed detection period. The fixed detection period can be 24 hours, that is, when the data processed by the graph construction module and the behavior coding module reaches the detection period, the detection module obtains abnormal behavior related abnormal processes from the global memory module.

[0052] According to an embodiment of the present application, the preset detection rule is that: first, a comprehensive scorer configured on the detection module is used to score each candidate process stored in the global memory module to obtain a comprehensive abnormal value of each candidate process; the comprehensive abnormal values of each candidate process are sorted in descending order, and a plurality of candidate processes greater than or equal to a preset second threshold and in front of the order are selected as abnormal processes. It should be noted that the second threshold can be determined according to actual needs, and the present application does not make specific limitations. For example, the second threshold can be set as β (0<β<1).

[0053] According to an embodiment of the present application, the comprehensive scorer is configured to score each candidate process according to the following steps. Each step is described below.

[0054] First, the rarity score of the key abnormal set corresponding to the candidate process is calculated as follows:

[0055]

[0056] Where KS κrarity score of a key anomaly set κ; κ represents a key anomaly set; ln(·) represents a logarithm function; min(·) represents a minimization function; |W| represents the number of received time windows; the number of time windows in which the key anomaly set κ exists, such as, for a specified key anomaly set κ1, assuming that there are processes p1 and p2 that hit the key anomaly set κ1 in a time window t1, and process p3 that hits the key anomaly set κ1 in a time window t2, then the specified key anomaly set κ1 exists in the time windows t1 and t2, and at this time the number of times that the key anomaly set κ is hit, such as, for a specified key anomaly set κ1, which is hit by the three processes p1, p2 and p3, at this time the maximum value among the number of times that each key anomaly set is hit in all key anomaly sets, such as, for a key anomaly set κ1, for a key anomaly set κ2, for a key anomaly set κ3, wherein the key anomaly set κ2 corresponds to the maximum, then the rarity score of the candidate process hitting the key anomaly sets κ1, κ2 and κ3 is calculated as follows:

[0057] Then, the rarity score of the candidate process hitting the key anomaly set is calculated as follows:

[0058]

[0059] wherein PS p⊥κ the rarity score of the candidate process p hitting the key anomaly set κ, p⊥κ represents the candidate process p hitting the key anomaly set κ (hit means that the intersection ratio between the key anomaly set corresponding to the process and other key anomaly sets is greater than or equal to a preset value); e represents an element in the key anomaly set κ; N(p) represents a neighbor set of the candidate process p; the number of times that the element e in the key anomaly set κ is hit, such as, for a specified key anomaly set κ1, which is hit by the processes p1, p2 and p3, the number of times that the element e in the key anomaly set κ1 is hit refers to the number of elements in the key anomaly set corresponding to the processes p1, p2 and p3 that are consistent with the element e.

[0060] Finally, the comprehensive anomaly value of the candidate process is calculated as follows:

[0061]

[0062] AS p⊥κ = RE p × KS κXPS p⊥κ

[0063] wherein, AS p denotes the abnormal value of the candidate process p; K p denotes the set composed of all the key abnormal set hit by the candidate process p; AS p⊥κ denotes the abnormal value of the key abnormal set K hit by the candidate process p, RE p denotes the abnormal score of the candidate process p based on the deviation.

[0064] It should be noted that the abnormal processes obtained based on the foregoing embodiment screening meet the conditions of large deviation error and rare behavior at the same time, and thus the abnormal processes obtained through screening can better determine which information is related to threats.

[0065] Five, an alarm module

[0066] The alarm module is configured to receive the abnormal processes transmitted by the detection module, read the global traceability graph and the key abnormal sets corresponding to all the abnormal processes from the global memory module according to all the received abnormal processes, reconstruct an abnormal traceability graph according to the mutual relationship of all the abnormal processes in the global traceability graph, and process the abnormal traceability graph according to a preset identification rule to output an abnormal behavior graph alarm. It should be noted that when the abnormal traceability graph is reconstructed, the direction of the edge can be defined as the direction of the dependency flow, wherein the dependency flow includes a control flow and a data flow. The control flow exists in process forking, for example, process p1 forks process p2, and the direction of the control flow is from process p1 to process p2. The data flow exists in process access to a file or an IP, for example, process p1 reads file f, and the direction of the data flow is from f to p1.

[0067] According to one embodiment of the present application, the preset identification rule is that the abnormal traceability graph is divided into a plurality of non-overlapping connected graphs, and the abnormal value of each connected graph is calculated according to a preset calculation rule, wherein the preset calculation rule is:

[0068]

[0069] wherein, AS g denotes the abnormal value of the connected graph g; K g denotes the number of key abnormal sets contained in the connected graph g; AS p⊥κ denotes the abnormal value of the key abnormal set K hit by the candidate process p, and when the abnormal value of each connected graph is calculated, p also represents the first process of the connected graph g that hits the key abnormal set K; the Grubbs test algorithm is performed based on the abnormal value of each connected graph to identify the connected graph with an outlier abnormal value, and the connected graph is taken as an abnormal behavior graph alarm, wherein the execution process of the Grubbs test algorithm can be represented as follows:

[0070] AS alert = Outlier({AS g | g∈G candidate})

[0071] G alert = {g| AS g ∈ AS alert})

[0072] Wherein, Outlier(·) represents the Grubbs test method, AS alert represents the outlier connected graph set, G alert represents the reported abnormal behavior graph set. It should be noted that in addition to using the Grubbs test algorithm, K-Nearest Neighbors, Isolation Forest, Local Outlier Factor, etc. algorithms can also be used.

[0073] The intrusion detection system described in the foregoing embodiments can be used to perform intrusion detection. Based on this, the present application also proposes an intrusion detection method, which comprises the following steps: step S1, obtaining log data of a historical time window of a to-be-detected device; step S2, processing the log data of the historical time window of the to-be-detected device using the intrusion detection system as described in the foregoing embodiments. The specific intrusion detection process involved in step S2 is as follows: the graph construction module first receives log data of a preset time window and constructs a window trace graph; then, the behavior encoding module processes the window trace graph to obtain a plurality of candidate processes and their corresponding key anomaly sets, and transmits the plurality of candidate processes and their corresponding key anomaly sets to the global memory module for storage; the detection module judges whether the detection period is reached, and if the detection period is reached, the abnormal processes are filtered from the global memory module and output to the alarm module, and the abnormal behavior graph alarm is output after being processed by the alarm module. It should be noted that the to-be-detected device can be a Linux host, an Apple operating system, a Microsoft operating system, an Android operating system, etc. host or end system.

[0074] In order to better understand that the present application has better intrusion detection effect compared with the prior art, the present application sets up a comparative experiment to evaluate the intrusion detection performance of the prior art and the present application.

[0075] In the comparison experiment, the CADETS data set and the THEIA data set in the existing DARPA TC data set are used as test data to test the detection performance between the existing technology and the intrusion detection system proposed by the present application. The experimental results are shown in Tables 1, 2 and 3.

[0076] Table 1 shows the precision and recall rate of the intrusion detection system proposed by the present application in detecting abnormal processes, abnormal files and abnormal IPs on different data under different attack dates. As can be seen from Table 1, the intrusion detection system proposed by the present application has great advantages in recall rate and precision at the node level.

[0077] Table 2 shows the performance of the intrusion detection system proposed by the present application and the prior art on the same data segment. As shown in Table 2, the intrusion detection system proposed by the present application is superior to the prior art (ThreaTrace) in terms of precision and recall rate of detecting abnormal processes, abnormal files and abnormal IPs, regardless of which data set.

[0078] Table 3 shows the number of abnormal behavior graphs output by the intrusion detection system proposed by the present application and the prior art on the same data segment. In Table 3, "# abnormal graphs" refers to the number of abnormal behavior graphs output, and "other graphs" refers to the number of normal behavior graphs output. As shown in Table 3, compared with the prior art, the intrusion detection system proposed by the present application outputs fewer normal graphs and can use fewer abnormal behavior graphs to show information related to threat scenarios.

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] Table 3

[0084]

[0085] The present application has the following beneficial effects: (1) a scorer is set to evaluate the abnormal situation of the process node in the window trace graph based on deviation, and a candidate process is selected based on the evaluation of each process node, thereby distinguishing normal behavior nodes and abnormal behavior nodes, which is beneficial to reduce node redundancy; (2) a KAS constructor is set to construct a key abnormal set for each candidate process, so as to judge the abnormal information of the candidate process according to the hit frequency and other statistical quantities of the key abnormal set recorded in the runtime, and then find the nodes that are truly related to attacks, which is beneficial to reduce node redundancy; (3) an alarm module is set to selectively reconstruct edges through the abnormal information of the process, and divide and report abnormal behavior graphs through connectivity, so that closely related abnormal behaviors are retained in the same abnormal behavior graph, so as to clearly and comprehensively retain and present information related to threats.

[0086] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order, in fact, some of the steps can be performed concurrently, or even in a changed order, as long as the desired function can be achieved.

[0087] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0088] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium can also include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se.

[0089] Having described above several aspects of the application, any modifications and variations that fall within the scope of the described embodiments can be apparent to those skilled in the art. The foregoing description of embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the application be limited not with this detailed description.

Claims

1. An intrusion detection system for analyzing log data to implement intrusion detection, characterized in that: The system comprises: A graph construction module is used to receive log data from a preset time window and construct a window traceability graph based on it. The window traceability graph includes multiple nodes and multiple edges connecting two nodes. A node represents a process, file, or IP address that appears within the preset time window, and an edge represents the interaction between the two nodes it connects. The behavior coding module includes a scorer and a KAS constructor. The scorer is used to score the processes in the window traceability graph to obtain a deviation-based anomaly score for each process. The KAS constructor is used to select multiple candidate processes according to the deviation-based anomaly score of each process according to preset screening rules, and extract the key anomaly set corresponding to each candidate process. The scorer is an anomaly neural network model trained with the process and its neighbors as input and the deviation-based anomaly score of the process as output. The process neighbors refer to the objects operated by the process in the same time window. The scorer is trained as follows: Obtaining log data of multiple preset time windows to construct training data, wherein the training data includes multiple training samples, each training sample includes a process and all neighbors of the process in the same time window; Construct an initial model, which includes a feature extractor, a memory encoder, a decoder, and an outlier calculator. The feature extractor is used to extract the behavioral features of the process. The memory encoder is used to extract the behavioral features of all neighbors corresponding to the process. The decoder is used to reconstruct the behavioral features of the process based on the behavioral features of all neighbors extracted by the memory encoder to obtain a reconstructed feature. The outlier calculator is used to calculate the reconstruction error between the behavioral features of the process extracted by the feature extractor and the reconstructed features obtained by the decoder, and use the reconstruction error as the deviation-based anomaly score of the process. Taking the training sample as input and the deviation-based anomaly score of the process as output, the parameters of the anomaly neural network model are updated by minimizing the preset training target until the model converges, where the preset training target is: in, represents the reconstruction error, represents the mean square error loss function, Represents the reconstruction feature, Represents the behavioral characteristics of a process; The global memory module is used to store all candidate processes and their corresponding key anomaly sets transmitted by the behavior encoding module, and to construct a global traceability graph based on all stored candidate processes and their corresponding key anomaly sets; A detection module is used to screen abnormal processes from all candidate processes stored in the global memory module according to preset detection rules and a fixed detection cycle; An alarm module is used to receive the abnormal process transmitted by the detection module, and read the global traceability graph and the key exception sets corresponding to all abnormal processes from the global memory module according to all the received abnormal processes, and reconstruct the abnormal traceability graph according to the mutual relationship of all abnormal processes in the global traceability graph, and process the abnormal traceability graph according to the preset recognition rules to output an abnormal behavior graph alarm.

2. The system according to claim 1, wherein: The preset screening rules are: The deviation-based anomaly score of each process in the window traceability graph is sorted in descending order, and a plurality of processes that are greater than or equal to the first threshold and ranked first are selected as candidate processes based on a preset first threshold.

3. The system according to claim 2, characterized in that The KAS constructor is configured to extract the key exception set corresponding to each candidate process in the following manner: Obtain the neighbor set corresponding to the process based on the window traceability graph constructed based on the time window of the process, where the neighbor set includes processes, files, or IP addresses corresponding to other nodes that have a connection relationship with the node corresponding to the process; Extract the files and IP addresses in the neighbor set, and use a preset whitelist to filter the files and IP addresses in the extracted neighbor set that are consistent with the elements in the whitelist to obtain a key exception set corresponding to the process.

4. The system according to claim 3, characterized in that The preset detection rules are: The comprehensive scorer configured on the detection module is used to score each candidate process stored in the global memory module to obtain the comprehensive abnormality value of each candidate process; The comprehensive abnormality value of each candidate process is sorted in descending order, and a plurality of candidate processes whose values ​​are greater than or equal to the second threshold and ranked first are selected as abnormal processes based on a preset second threshold.

5. The system according to claim 4, characterized in that The comprehensive scorer is configured to score each candidate process as follows: The rarity score of the key anomaly set corresponding to the candidate process is calculated as follows: in, Represents a key exception set Rarity score of Represents a key exception set; represents the minimization function; represents the logarithmic function; Indicates the number of received time windows; Represents a key exception set The number of time windows that are hit; Represents a key exception set Number of hits; It represents the maximum number of times each key anomaly set is hit among all key anomaly sets; The rarity score of a candidate process hitting the critical anomaly set is calculated as follows: in, Represents a candidate process Hit key exception set The rarity score of Represents a candidate process Hit key exception set ; Represents a key exception set Elements in Represents a candidate process The set of neighbors of Represents a key exception set Chinese elements Number of hits; The composite outlier value of a candidate process is calculated as follows: in: in, Represents a candidate process The comprehensive outlier value of Represents a candidate process The set of all key exception sets that were hit; Represents a candidate process Hit key exception set The outliers, Represents a candidate process Deviation-based anomaly score.

6. The system according to claim 5, characterized in that The preset identification rules are: The anomaly tracing graph is divided into multiple non-overlapping connected graphs, and the anomaly value of each connected graph is calculated according to the preset calculation rules, where the preset calculation rules are: in, Represents a connected graph The outliers, Represents a connected graph The number of key anomaly sets included in ; Based on the outliers of each connectivity graph, the Grubbs test algorithm is executed to identify connectivity graphs with outliers and alarm them as abnormal behavior graphs.

7. An intrusion detection method, characterized in that: The method comprises: Step S1: Obtain log data of a historical time window of a device to be detected; Step S2: Process the log data of the historical time window of the device to be detected using the intrusion detection system according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to claim 7.

9. An electronic device, characterized in that: include: One or more processors, and memory, wherein the memory is configured to store executable instructions; The one or more processors are configured to implement the steps of the method of claim 7 by executing the executable instructions.

Images