Multi-source data fusion driven intrusion detection rule generation method

Abstract

The invention discloses a multi-source data fusion driven intrusion detection rule generation method, which comprises the following steps: acquiring three data sources, namely abnormal flow data, open source threat intelligence information and alarm log data, and sorting the quality of the open source threat intelligence information based on a graph database and a community detection algorithm to obtain high-quality intelligence. Alignment of abnormal traffic data and alarm logs is realized by adopting a three-layer model, and alignment of the abnormal traffic data, the alarm logs and the alarm logs is realized based on a decision tree model. And traversing the constructed decision tree to obtain a related path, and extracting an intrusion detection rule from the path. Multiple data sources based on abnormal flow data, alarm logs and threat intelligence information are organically combined, and intrusion detection rules with high timeliness can be generated. An improved sliding window + n-gram algorithm + cosine similarity three-layer alignment model solves the problem that alarm logs and abnormal traffic data are difficult to align due to network delay and system processing delay, and a credibility basis is provided for subsequent rule generation.

Description

Technical Field

[0001] The present invention relates to the field of network security (research direction of intrusion detection system), and in particular to an automatic intrusion detection rule generation method based on multi-source data fusion. Background Art

[0002] Mainstream intrusion detection systems include snort, suricata, etc., and their detection effects depend on the quality of the rule base.

[0003] Existing patent applications:

[0004] A network manufacturing system intrusion detection method, device and electronic equipment.

[0005] A network intrusion detection rule generation method, apparatus, device and storage medium.

[0006] A method and system for generating intrusion detection rules based on honeypot attack alarms.

[0007] Intrusion detection rule generation method, device and electronic equipment.

[0008] Published papers:

[0009] A Conceptual Framework for Automated Rule Generation in Provenance-based Intrusion Detection Systems.

[0010] A method for automatically generating intrusion detection rules based on threat intelligence.

[0011] Dendron: Genetic trees driven rule induction for network intrusion detection systems.

[0012] Under the current situation, the methods of network attacks are endless, the attack capabilities are constantly increasing, and the defense capabilities need to be improved urgently. The P2DR model proposes that under the guidance of security strategies, the three modules of protection, detection and response are linked together to form a closed loop to ensure the security of information systems. In the protection module, the implementation of software and hardware firewalls is relatively mature; in the detection module, intrusion detection systems (IDS) such as Snort are also relatively mature. Protection and detection as defense capabilities, combined with the response module to link other network security protection devices to deal with threats, ensure the security of information systems.

[0013] At present, the detection capability of intrusion detection systems depends on the quality of their rule base. The extraction and update of rule feature bases mainly rely on security experts and software implementation, but this manual collection method is costly and inefficient. In terms of automatic generation and update of rule bases, some researchers have proposed intrusion detection rule generation algorithms and devices for video surveillance equipment. They use the principle that video surveillance equipment will not send requests to other devices in principle, reverse trace abnormal requests, and generate intrusion detection rules, but they are limited to the field of video surveillance and are not universal. Some researchers have proposed using honeypots to collect alarm data, and using differential analysis to generate differential results, and further generate intrusion detection rules, but their data sources are single, and the quality of generation is highly dependent on the quality of honeypot collection. Other researchers have used a large amount of historical data to build a knowledge graph, extract traffic features from the abnormal traffic of the captured target system, and then determine the intrusion detection rules from the knowledge graph, but this method is complicated to build the graph and is not very timely. Some researchers have also recorded the normality to reverse the abnormality based on the fact that the data flow direction in the numerical control field is fixed, and generated intrusion detection rules in the numerical control field based on this. This method also has the problem of low universality. Summary of the invention

[0014] In summary, the manual collection method is costly and inefficient, and some existing methods for automatically generating intrusion detection rules have problems of low universality or timeliness. At the same time, we note that the timeliness of threat intelligence is higher than the above-mentioned IDS rules that are manually constructed or based on historical data. Therefore, the present invention proposes a multi-source fusion method that can organically combine network traffic data, alarm logs, and threat intelligence data to improve the generation efficiency and timeliness of intrusion detection rules.

[0015] In view of this, the technical solution adopted by the present invention is an intrusion detection rule generation method driven by multi-source data fusion, which collects three data sources: abnormal traffic data, open source threat intelligence information, and alarm log data, and sorts the quality of the open source threat intelligence information based on the graph database and community detection algorithm to obtain high-quality intelligence. Then, a three-layer model is used to realize the alignment of abnormal traffic data and alarm logs, and the alignment of the three is further realized based on the decision tree model. The constructed decision tree is traversed to obtain the relevant path, and the intrusion detection rules can be extracted from the path.

[0016] The specific steps are as follows:

[0017] Step 1. Collect the raw data used to generate intrusion detection rules, including abnormal traffic data of a given target network, open source threat intelligence information in the Internet space, and alarm logs obtained after traffic passes through the intrusion detection system. Go to step 2.

[0018] Step 2. There is a certain time difference between the abnormal traffic data obtained in step 1 and the alarm log, and the alarm log is the data obtained after the abnormal traffic data flows to the intrusion detection system. It can be seen that the number of alarm logs is less than or equal to the abnormal traffic data, and it is impossible to obtain the alignment effect one by one. The improved sliding window algorithm is used to obtain the abnormal traffic data set within a period of time corresponding to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 3.

[0019] Step 3. Extract the main feature fields of the abnormal traffic data and alarm log obtained in step 2, which are expressed in text form, and calculate the similarity between the alarm log and each piece of data in its corresponding abnormal traffic data set through the N-gram similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 4.

[0020] Step 4. Extract the main feature fields of the abnormal traffic data set and the alarm log in step 3, which are expressed in text form, and calculate the similarity between the alarm log and each piece of data in its corresponding abnormal traffic data set through the cosine similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 5.

[0021] Step 5. Randomly select an abnormal flow from the abnormal flow data set and input it to step 6.

[0022] Step 6. Take out the field information not included in the abnormal traffic data from the alarm log, and append it to the abnormal traffic data to obtain a new data format, so as to align the field information of the two. Go to step 7.

[0023] Step 7. Write the threat intelligence information obtained in step 1 into the graph database as the input of the graph database, store different relationships between intelligence, and assign different weights according to the importance of the relationship. Execute step 8.

[0024] Step 8. Use the threat intelligence information after weighting in step 7 as the input of the community detection algorithm, and divide the community based on the idea of ​​maximizing modularity. And calculate the intelligence quality of the community based on the average value of the intelligence quality in the community. For a single intelligence, its intelligence quality is the sum of the average quality of the community and the quality of its own intelligence source, and sort the intelligence to obtain a collection of high-quality intelligence. Execute step 9.

[0025] Step 9. Initialize the decision tree model. At the root node of the decision tree, select a key field (such as source IP) from the aligned traffic and log data obtained in step 6 for splitting. Then recursively select other fields in the child nodes, such as destination IP, port number, etc., and gradually build the structure of the decision tree. In each node or path of the decision tree, introduce the threat intelligence field associated with the current node field. For example, if the current node judges the source IP field, the source IP field in the threat intelligence is also checked when judging. When a path of the decision tree contains a field match related to threat intelligence, these intelligence fields are merged into the current path, and the intelligence quality obtained in step 8 is also added to the field. Execute step 10.

[0026] Step 10. By traversing the decision tree, the paths containing traffic, logs, and threat intelligence are extracted. Each path represents a fused data entry, containing relevant information about traffic, logs, and intelligence. The relevant information is filled in according to the format of the intrusion detection system rules to obtain the intrusion detection rules.

[0027] The present invention adopts multiple data sources based on abnormal traffic data, alarm logs, and threat intelligence information, and uses the three to organically combine them to generate intrusion detection rules with strong timeliness.

[0028] The improved sliding window + n-gram algorithm + cosine similarity three-layer alignment model in the present invention solves the problem of difficulty in aligning alarm logs and abnormal traffic data caused by network delays and system processing delays.

[0029] In view of the problem that the quality of existing open source threat intelligence sources on the Internet varies, the present invention proposes a division method based on a community detection algorithm. By calculating the quality of the average intelligence within the community, the division of high-quality intelligence can be obtained, which also provides a credible basis for the generation of subsequent rules.

[0030] The present invention adopts a decision tree to fuse multi-source data, and can further efficiently convert it into intrusion detection rules by traversing the decision tree access path. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0032] Figure 1 A schematic diagram of the process of the intrusion detection rule generation method driven by multi-source data fusion according to the present invention;

[0033] Figure 2 This is a schematic diagram of threat intelligence data acquisition in step 1 of the first embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of a three-layer alignment model in steps 2 to 5 of the first embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of performing community quality sorting based on a community detection algorithm in step 8 of the first embodiment of the present invention; DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0037] Embodiment 1

[0038] like Figure 1 As shown, the first embodiment is an intrusion detection rule method driven by multi-source data fusion, which consists of 10 steps. The specific description is as follows:

[0039] Step 1. Collect the raw data used to generate intrusion detection rules, including 20,000 pieces of abnormal traffic data of a given target network, 10,000 pieces of open source threat intelligence information in the Internet space, and 3,000 alarm logs obtained after the traffic passes through the intrusion detection system. Go to step 2.

[0040] Step 2. There is a certain time difference between the abnormal traffic data obtained in step 1 and the alarm log, and the alarm log is the data obtained after the abnormal traffic data flows to the intrusion detection system. It can be seen that the number of alarm logs is less than or equal to the abnormal traffic data, and it is impossible to obtain the alignment effect one by one. Therefore, the improved sliding window algorithm is used to obtain the abnormal traffic data set within a period of time corresponding to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 3.

[0041] Step 3. Extract the main feature fields of the abnormal traffic data and alarm log obtained in step 2, which are expressed in text form, and calculate the similarity between the alarm log and each piece of data in its corresponding abnormal traffic data set through the N-gram similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 4.

[0042] Step 4. Extract the main feature fields of the abnormal traffic data set and the alarm log in step 3, which are expressed in text form. Calculate the similarity of each data in the alarm log and its corresponding abnormal traffic data set through the cosine similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log. If the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 5.

[0043] Step 5. Randomly select an abnormal flow from the abnormal flow data set and input it to step 6.

[0044] Step 6. Take out the field information not included in the abnormal traffic data from the alarm log, and append it to the abnormal traffic data to obtain 3000 new data forms, so as to align the field information of the two. Go to step 7.

[0045] Step 7. Write the threat intelligence information obtained in step 1 into the graph database as the input of the graph database, store different relationships between intelligence, and assign different weights according to the importance of the relationship. Execute step 8.

[0046] Step 8. Use the threat intelligence information after weighting in step 7 as the input of the community detection algorithm, and divide the community based on the idea of ​​maximizing modularity, and obtain 256 community divisions. And calculate the intelligence quality of the community based on the average value of the intelligence quality in the community. For a single intelligence, its intelligence quality is the sum of the average quality of the community and the quality of its own intelligence source, and sort the intelligence to obtain a collection of high-quality intelligence. Execute step 9.

[0047] Step 9. Initialize the decision tree model. At the root node of the decision tree, select a key field (such as source IP) from the aligned traffic and log data obtained in step 6 for splitting. Then recursively select other fields in the child nodes, such as destination IP, port number, etc., and gradually build the structure of the decision tree. In each node or path of the decision tree, introduce the threat intelligence field associated with the current node field. For example, if the current node judges the source IP field, the source IP field in the threat intelligence is also checked when judging. When a path of the decision tree contains a field match related to threat intelligence, these intelligence fields are merged into the current path, and the intelligence quality obtained in step 8 is also added to the field. Execute step 10.

[0048] Step 10. By traversing the decision tree, the paths containing traffic, logs, and threat intelligence are extracted. Each path represents a fused data entry, containing relevant information about traffic, logs, and intelligence. The relevant information is filled in according to the format of the intrusion detection system rules, and 1576 intrusion detection rules are obtained.

[0049] Figure 1 A schematic diagram of the process of the intrusion detection rule generation method driven by multi-source data fusion according to the present invention;

[0050] Figure 2 This is a schematic diagram of threat intelligence data acquisition in step 1 of the first embodiment of the present invention;

[0051] like Figure 2As shown, the Internet space contains a variety of open source threat intelligence information, including but not limited to machine-readable intelligence, human-readable intelligence, portrait intelligence and other forms of expression. For machine-readable intelligence, by calling the network interface to obtain JSON data in a unified format, threat data that can be directly parsed by the machine is obtained (such as one or more of the IP address, threat type, malicious identification field, degree of harm, and domain name reverse query information). Human-readable intelligence mainly extracts key information from unstructured data such as open source blogs (i.e. web pages), such as IOC data, attack characteristics, payload parameters, etc. In addition, some additional supplementary information, such as attack paths, can be obtained by collecting portrait intelligence. The characteristic fields of the three types of intelligence are unified, specifically, the union of the fields of the three types of intelligence is first taken, and then the duplicate fields are removed to realize the processing of intelligence.

[0052] Figure 3 This is a schematic diagram of a three-layer alignment model in steps 2 to 5 of the first embodiment of the present invention;

[0053] Alignment model such as Figure 3 As shown in the figure, a three-layer alignment model is used: improved sliding window + N-gram similarity + cosine distance to align traffic data and alarm logs.

[0054] Considering that the original format of traffic data is a binary stream file, it does not contain an identifier that can effectively identify the traffic sequence, namely the traffic ID, and cannot be directly mapped with the alarm log through the ID. In addition, the timestamp field that can be used as the basis for mapping will be affected by network fluctuations, intrusion detection rule base call efficiency, and other issues, and there is no possibility of direct and complete matching. Therefore, an improved sliding window algorithm is used to use the threshold as the boundary to adaptively change the window size for the abnormal traffic density corresponding to a specific alarm day time window, and the abnormal traffic in the time window is used as the alignment traffic candidate for the alarm log.

[0055] The abnormal traffic obtained after the sliding window is a set sequence. At this time, it is impossible to align the traffic with the alarm log one-to-one. Further, from the candidate traffic obtained from the sliding window, the N-gram similarity algorithm is used to calculate the similarity between the traffic and the candidate alarm log, and sort them. If there is only one traffic with the highest similarity, the field alignment operation is performed, that is, the same field is removed and different fields are appended. If there are N traffics with the highest similarity, the cosine distance is used for further calculation, and the word frequency is calculated to convert it into numerical data, and then further sorted by the cosine distance. If a matching traffic is found, it is returned for alignment operation. If there are M traffics that meet the requirements, one is randomly selected.

[0056] Figure 4 This is a schematic diagram of performing community quality sorting based on a community detection algorithm in step 8 of the first embodiment of the present invention;

[0057] The quality of intelligence on the Internet varies. Some intelligence sources have reliable historical observations, while others are difficult to distinguish. Considering that the quality of the same intelligence source varies, a community detection algorithm is used to classify the quality of different intelligence sources.

[0058] First, the threat intelligence is written into the graph database as the input of the graph database, and the relationship between the intelligence is stored. Specifically, the relationship types are defined as: irrelevant, similar data sources, same data sources, similar features, and same features. The graph database maintains the relationship between different intelligence and calculates the weights between each of them based on the relationship between them.

[0059] The weighted relationship graph is input into the graph analysis algorithm - community detection algorithm, and different community divisions are obtained through multiple iterations. The community division is based on the idea of ​​maximizing modularity, which can ensure the maximum module benefit in each local move. The calculation formula of modularity is as follows:

[0060]

[0061] in:

[0062] ·

[0063] Let Aij be the weight of the edge between node i and node j. If it is an unweighted graph, Aij is either 1 (edge) or 0 (no edge).

[0064] Let ki be the degree (or weight sum) of node i.

[0065] Let m be the total weight of all edges in the graph.

[0066] ·Let ci be the community to which node i belongs.

[0067] Let δ(ci,cj) be an indicator function that is 1 if ci = cj and 0 otherwise.

[0068] For the same community, its internal intelligence has an initial quality coefficient, where the initial quality coefficient comes from the historical performance and recognition of the intelligence source. Taking into account the different amounts of community intelligence, the quality of the community is determined by calculating the average value of the community intelligence quality coefficient. For a single intelligence, its intelligence quality is the sum of the average quality of the community and the quality of its own intelligence source, and the intelligence is sorted to obtain a collection of high-quality intelligence. The intelligence quality calculation formula is as follows:

[0069]

[0070]

[0071] Among them: Let Ci represent the i-th community, which contains ni nodes (intelligence);

[0072] Let wij represent the quality of the intelligence source of the jth node in community Ci;

[0073] Let qij denote the final quality of the j-th node in community Ci.

Claims

1. A method for generating intrusion detection rules driven by multi-source data fusion, characterized in that: Collect three data sources: abnormal traffic data, open source threat intelligence information, and alarm log data, and sort the quality of the open source threat intelligence information based on the graph database and community detection algorithm to obtain high-quality intelligence; then use a three-layer model to align the abnormal traffic data with the alarm log, and align the three based on the decision tree model; traverse the constructed decision tree to obtain the relevant path, and extract the intrusion detection rules from the path; The specific steps are as follows: Step 1. Collect the raw data used to generate intrusion detection rules, including abnormal traffic data of a given target network, open source threat intelligence information in the Internet space, and alarm logs obtained after traffic passes through the intrusion detection system; proceed to step 2; Step 2. There is a certain time difference between the abnormal traffic data obtained in step 1 and the alarm log, and the alarm log is the data obtained after the abnormal traffic data flows to the intrusion detection system. It can be seen that the number of alarm logs is less than or equal to the abnormal traffic data, and it is impossible to obtain the alignment effect one by one. The improved sliding window algorithm is used to obtain the abnormal traffic data set within a period of time corresponding to the alarm log; if the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 3; Step 3. Extract the main feature fields of the abnormal traffic data and alarm log obtained in step 2, which are expressed in text form, calculate the similarity of each data of the alarm log and its corresponding abnormal traffic data set through the N-gram similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log; if the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 4; Step 4. Extract the characteristic fields of the abnormal traffic data set and the alarm log in step 3, which are expressed in text form, calculate the similarity of each data of the alarm log and its corresponding abnormal traffic data set by the cosine similarity algorithm, sort them based on the similarity, and obtain the abnormal traffic data set with the greatest similarity to the alarm log; if the amount of abnormal traffic data is 1, execute step 6, otherwise execute step 5; Step 5. Randomly select an abnormal flow from the abnormal flow data set and input it to step 6; Step 6. Take out the field information not included in the abnormal traffic data from the alarm log, and append it to the abnormal traffic data to obtain a new data format, so as to align the field information of the two; execute step 7; Step 7. Write the threat intelligence information obtained in step 1 into the graph database as the input of the graph database, store different relationships between the intelligence, and assign different weights according to the importance of the relationship; execute step 8; Step 8. Use the threat intelligence information after weighting in step 7 as the input of the community detection algorithm, and divide the community based on the idea of ​​maximizing modularity; and calculate the intelligence quality of the community based on the average value of the intelligence quality in the community; For a single intelligence, its intelligence quality is the sum of the average quality of the community and the quality of its own intelligence source, and the intelligence is sorted to obtain a collection of high-quality intelligence; execute step 9; Step 9. Initialize the decision tree model. At the root node of the decision tree, select a key field from the aligned traffic and log data obtained in step 6 for splitting; then recursively select the destination IP and port number in the child nodes to gradually build the structure of the decision tree; in each node or path of the decision tree, introduce the threat intelligence field associated with the current node field; when a path of the decision tree contains a field match related to threat intelligence, merge these intelligence fields into the current path, and the intelligence quality obtained in step 8 will also be added to the field; execute step 10; Step 10. By traversing the decision tree, the path containing traffic, logs, and threat intelligence is extracted; Each path represents a fused data entry, containing information about traffic, logs, and intelligence; Fill in the relevant information according to the format of the intrusion detection system rules to obtain the intrusion detection rules.

2. The multi-source data fusion driven intrusion detection rule generation method according to claim 1 is characterized in that: The three-layer alignment model includes a sliding window, an n-gram algorithm and cosine similarity.

3. The intrusion detection rule generation method driven by multi-source data fusion according to claim 1 is characterized in that: The Internet space contains a variety of open source threat intelligence information, including machine-readable intelligence, human-readable intelligence, and portrait intelligence. For machine-readable intelligence, the network interface is called to obtain JSON data in a unified format to obtain threat data that can be directly parsed by the machine. Human-readable intelligence is the extraction of key information from unstructured data such as open source blogs.

4. The method for generating intrusion detection rules driven by multi-source data fusion according to claim 1, characterized in that: Adopting an improved sliding window algorithm, for the abnormal traffic density corresponding to a specific alarm day time window, using the threshold as the boundary, adaptively changing the window size, and taking the abnormal traffic in the time window as the alignment traffic candidate of the alarm log; The abnormal traffic obtained after the sliding window is a set sequence. At this time, it is impossible to align the traffic with the alarm log one-to-one. Further, from the candidate traffic obtained from the sliding window, the N-gram similarity algorithm is used to calculate the similarity between the traffic and the candidate alarm log, and sort them. If there is only one traffic with the highest similarity, a field alignment operation is performed, that is, the same fields are removed and different fields are appended. If there are N traffics with the highest similarity, the cosine distance is used for further calculation, and the word frequency is calculated to convert it into numerical data, and further sorted by the cosine distance. If a matching traffic is found, it is returned for alignment operation. If there are M traffics that meet the requirements, one is randomly selected.

5. The method for generating intrusion detection rules driven by multi-source data fusion according to claim 1, characterized in that: A community detection algorithm is used to classify the quality of different intelligence.

6. The method for generating intrusion detection rules driven by multi-source data fusion according to claim 1, characterized in that: First, the threat intelligence is written into the graph database as the input of the graph database, and the relationship between the intelligence is stored. The relationship types are defined as follows: irrelevant, similar data sources, same data sources, similar features, and same features. The graph database maintains the relationship between different intelligence and calculates the weights between each of them based on the relationship between them; The weighted relationship graph is input into the graph analysis algorithm - community detection algorithm, and different community divisions are obtained through multiple iterations. The community division is based on the idea of ​​maximizing modularity, which can ensure the maximum module benefit in each local move. The calculation formula of modularity is as follows: Where: Aij is the weight of the edge between node i and node j; if it is an unweighted graph, Aij is either 1 with an edge or 0 without an edge; ki is the degree or weight sum of node i; m is the total weight of all edges in the graph; ci is the community to which node i belongs; δ(ci,cj) is an indicator function, which is 1 if ci = cj and 0 otherwise.

7. The multi-source data fusion driven intrusion detection rule generation method according to claim 1, characterized in that: For a single piece of intelligence, its quality is the sum of the average quality of the community and the quality of its own intelligence source. The intelligence is sorted to obtain a collection of high-quality intelligence. The formula for calculating intelligence quality is as follows: Where: Let Ci represent the i-th community, which contains ni nodes; Let wij represent the quality of the intelligence source of the jth node in community Ci; Let qij denote the final quality of the j-th node in community Ci.

Images