A Comprehensive Quality Dynamic Evaluation Method and System for Cyber Threat Intelligence

Abstract

The present invention discloses a method and system for dynamically evaluating the comprehensive quality of network threat intelligence, including: Step 1, structurally defining network threat intelligence; Step 2, evaluating the quality of intelligence sources: using an iterative algorithm for evaluating the quality of intelligence sources to evaluate the content authority and link authority of intelligence sources; Step 3, evaluating the quality of intelligence content: for two pieces of intelligence with the same IP value or the same domain name, calculating their similarity based on four features: intelligence source, timestamp, threat category, and description tags; Step 4, comprehensively and dynamically evaluating the intelligence quality. The method for dynamically evaluating the comprehensive quality of network threat intelligence provided by the present invention comprehensively considers multiple indicators such as intelligence sources and intelligence content, and evaluates the intelligence quality from multiple dimensions.

Description

Technical Field

[0001] The present invention belongs to the technical field of cyber threat intelligence, and relates to a method and system for dynamically evaluating the comprehensive quality of cyber threat intelligence. Background Art

[0002] In recent years, with the rapid development and progress of related technologies, cyber space technology has been continuously evolving and upgrading. In the cyber security battlefield, the attacker is in the dark while we are in the open. The attacker's attack methods are constantly changing, the attack tools are rich and efficient, the attack cost is relatively low, and the preparation time is relatively long. Nowadays, a new generation of attackers often launch targeted cyber attacks on enterprises and organizations, namely "Advanced Persistent Threats" (APTs). Such attacks on specific enterprises or industries are generally carefully planned, with intricate attack methods, and often result in serious data leakage or damage. Most traditional security protection methods are passive, relying on security devices for static defense. In the face of new threats and attacks, the effects of defense and detection are minimal. How to change from suddenly encountering problems and then emergently solving them to foreseeing problems and taking preventive measures in advance has become one of the urgent needs of key industries.

[0003] Under such circumstances, in order to meet the requirements of proactive prevention, improve the detection and recognition accuracy, shorten the response time, and reduce the defense cost, cyber threat intelligence has emerged. Generally speaking, threat intelligence refers to knowledge that can be used to address threats or respond to hazards, including threat sources, attack intentions, attack techniques, and attack target information. It has characteristics such as high knowledge density, high accuracy, and strong relevance, and can provide strong data support for all stages of security analysis, and can respond to polymorphic and complex highly intelligent threats and attacks in a timely manner.

[0004] Threat intelligence is used to assist in supporting decision-making or security analysis. Therefore, intelligence with unknown sources and authenticity will affect the correctness of decision-making and the accuracy of analysis results. There may be multiple sources for threat intelligence of the same category, such as internal security event information in the target system, commercial threat intelligence, open-source threat intelligence, etc. The data quality is uneven. Therefore, it is necessary to screen and evaluate the intelligence quality to obtain high-quality and highly credible intelligence. How to scientifically and objectively evaluate the quality of multi-source threat intelligence has an urgent practical significance for improving the national network security guarantee ability. Under the extremely high application value and considerable market prospects, threat intelligence has been facing the problem of unclear quality assessment standards and methods. The existing research work is partly qualitative evaluation and mostly quantitative evaluation. They start from the perspective of intelligence sources or from the perspective of intelligence content characteristics, and apply technologies such as machine learning or graph mining to evaluate the quality of threat intelligence. However, it is incomplete to only consider these dimensions separately when evaluating the quality of threat intelligence. In addition, few studies have paid attention to the problem of time dynamics and the mutual influence of the quality of homologous intelligence. How to construct a network threat intelligence quality assessment scheme that integrates various dimensions, multiple technical methods, and is applicable to the dynamic changes of the network security environment is an urgent problem to be solved in the current field of network threat intelligence quality assessment. Summary of the Invention

[0005] The purpose of the present invention is to provide a comprehensive quality dynamic assessment method and system for network threat intelligence, which solves the problems of single evaluation dimension, isolated evaluation technology, and ignoring the dynamic changes of the network security environment in the prior art.

[0006] The technical solution adopted by the present invention is as follows.

[0007] A comprehensive quality dynamic assessment method for network threat intelligence includes:

[0008] Step 1: Structurally define network threat intelligence:

[0009] CyberThreatIntelligence = (source, time, category, tag, geo, value);

[0010] Among them, CyberThreatIntelligence represents network threat intelligence, source represents the intelligence source, time represents the intelligence timestamp, category represents the threat type, tag represents the threat description label, geo represents the geographical location of the intelligence IP, and value represents the specific IP value or domain name;

[0011] Step 2: Evaluate the quality of the intelligence source: Use the intelligence source quality evaluation iterative algorithm to evaluate the content authority and link authority of the intelligence source.

[0012] Step 3. Intelligence Content Quality Assessment: For two pieces of intelligence with the same IP value or the same domain name, calculate their similarity based on four features: intelligence source, timestamp, threat category, and description tag.

[0013] Step 4. Comprehensive Dynamic Assessment of Intelligence Quality: For each piece of threat intelligence, the quality based on the intelligence source obtained from Step 2 and the quality based on the intelligence content obtained from Step 3 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous piece of threat intelligence from the same source and introducing a time decay factor to weaken the influence of intelligence with a long time interval, the comprehensive quality score of the nth intelligence sample is obtained.

[0014] Optionally, the specific steps of Step 2 include:

[0015] The input of the intelligence source quality assessment iterative algorithm is the intelligence source relationship graph G, G=(V, E, W), where each node v∈V represents an intelligence source; when the IP value or DNS of the intelligence i generated by the intelligence source u is the same as that of the intelligence j generated by the intelligence source v, and the timestamp of i is later than that of j, two nodes u, v∈V are connected by a directed edge (u, v)∈E, indicating that the intelligence source u quotes the intelligence of the intelligence source v; the weight of the edge (u, v) is the number of times w∈W is quoted.

[0016] Initialize the content authority vector A and the link authority vector H, and set the lengths of both vector A and vector H to 1; in the kth iteration step, the authority value of the intelligence source S is calculated by and the hub value of the intelligence source S is calculated by ; Normalize the vectors A and H, and iterate in a loop until convergence; finally, output the authority values of each intelligence source, and convert the authority values of each intelligence source into a percentage system as the quality score s of each intelligence i based on the intelligence source source ; w i represents the weight of the corresponding edge, and n represents the number of connected nodes.

[0017] Optionally, the specific steps of Step 3 include:

[0018] S(v t , v i ) = θ1×S source + θ2×S time + θ3×S category + θ4×S tag ;

[0019] s.t. θ1 + θ2 + θ3 + θ4 = 1; s.t. means subject to;

[0020] where, S(vt , v i ) represents the similarity of two threat intelligence, S source represents the similarity of information sources, S time is the time similarity of threat intelligence, S category is the similarity of threat categories, S tag is the similarity of threat description tags; θ1, θ2, θ3, θ4 are the weights of these four factors respectively. In the experiment, the influences of various factors are considered equally, and the value of each weight is 0.25; for two completely identical pieces of intelligence, their similarity is the highest, with a value of 1.

[0021] Optionally, the calculation methods for the information source similarity and threat type similarity are as follows: when the sources of two threat intelligence are the same, the information source similarity S source takes 1, and when they are different, S source takes 0; the value of the threat category similarity S category is obtained in the same way.

[0022] Optionally, the calculation method for the time similarity includes:

[0023]

[0024] First, calculate the value of the time difference between the intelligence release times |t(v t , v i ), and then normalize it to obtain the time distance Finally, calculate the time similarity S time (v t , v i ), with a value range of [0, 1]; if the timestamps t of two threat intelligence are the same, then their time distance is 0, and the time similarity S time is 1.

[0025] Optionally, the calculation of the similarity of threat description tags includes:

[0026]

[0027] In the formula, X t and X i are respectively the vector representations of the threat description tags of threat intelligence v t and v i , is the cosine similarity of the two vectors, with a value range of [0, 1]; when the threat intelligence tags are the same, the cosine similarity is 1; if the threat tags tag of both threat intelligence are empty, it is stipulated that the similarity between them is 0.5.

[0028] Optionally, it further includes constructing a quality evaluation index based on intelligence content. The preprocessed feature data is used as the input of the classifier. The KNN algorithm is applied to classify intelligence samples into five categories according to the credibility level, and each category is assigned an intelligence quality score s based on the content. content 。

[0029] Optionally, step 4 specifically includes:

[0030]

[0031] Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence homologous to the nth intelligence sample; D represents the time decay factor; s i (n) represents the ith basic quality score of this intelligence sample; w i is the weight of each factor s i (n).

[0032] A network threat intelligence comprehensive quality dynamic evaluation system, which is used to run any of the network threat intelligence comprehensive quality dynamic evaluation methods of the present invention;

[0033] It includes: an intelligence source quality evaluation module: using an intelligence source quality evaluation iterative algorithm to evaluate the content authority and link authority of the intelligence source;

[0034] An intelligence content quality evaluation module: For two pieces of intelligence with the same IP value or the same domain name, calculate their similarity based on four features: the intelligence source, timestamp, threat category, and description label;

[0035] An intelligence quality comprehensive dynamic evaluation module: For each piece of threat intelligence, the quality based on the intelligence source obtained from step 2 and the quality based on the intelligence content obtained from step 3 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous threat intelligence homologous to it and introducing a time decay factor to weaken the influence of intelligence with a long time interval, the comprehensive quality score of the nth intelligence sample is obtained.

[0036] Optionally, the information source quality assessment module: uses the information source quality assessment iterative algorithm to evaluate the content authority and link authority of the information source; the input of the information source quality assessment iterative algorithm is the information source relationship graph G, G = (V, E, W), where each node v ∈ V represents an information source; when the IP value or DNS of the information i generated by the information source u is the same as that of the information j generated by the information source v, and the timestamp of i is later than that of j, two nodes u, v ∈ V are connected by a directed edge (u, v) ∈ E, indicating that the information source u quotes the information of the information source v; the weight of the edge (u, v) is the number of times w ∈ W is quoted;

[0037] Initialize the content authority vector A and the link authority vector H, and set the lengths of both the vector A and the vector H to 1; in the k-th iteration, the authority value of the information source S is calculated by and the hub value of the information source S is calculated by ; normalize the vectors A and H, and iterate in a loop until convergence; finally, output the authority values of each information source, and convert the authority values of each information source into a percentage system as the quality score s of each information i based on the information source source ; w i represents the weight of the corresponding edge, and n represents the number of connected nodes;

[0038] The information content quality assessment module: calculates the similarity of two pieces of information with the same IP value or the same domain name based on four features: information source, timestamp, threat category, and description label;

[0039] S(v t , v i ) = θ1 × S source + θ2 × S time + θ3 × S category + θ4 × S tag ;

[0040] s.t. θ1 + θ2 + θ3 + θ4 = 1; s.t. means subject to;

[0041] where S(v t , v i ) represents the similarity of two threat intelligence, S source represents the similarity of the information source, S time is the time similarity of the threat intelligence, S category is the similarity of the threat category, S tag is the similarity of the threat description label; θ1, θ2, θ3, θ4 are the weights of each of these four factors. In the experiment, the influence of each factor is considered equally, and the value of each weight is 0.25; for two completely identical pieces of information, their similarity is the highest, with a value of 1;

[0042] Comprehensive Dynamic Evaluation Module for Intelligence Quality:

[0043] Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence homologous to the nth intelligence sample; D represents the time decay factor; s i (n) represents the i-th basic quality score of this intelligence sample; w i is the weight of each factor s i (n).

[0044] The beneficial effects of the present invention are:

[0045] 1. The comprehensive dynamic evaluation method for network threat intelligence provided by the present invention comprehensively considers multiple indicators such as intelligence sources and intelligence content, and evaluates intelligence quality from multiple dimensions;

[0046] 2. The comprehensive dynamic evaluation method for network threat intelligence provided by the present invention combines multiple methods such as graph mining, machine learning, and weighted average models, and fully utilizes the advantages of each technical method;

[0047] 3. The comprehensive dynamic evaluation method for network threat intelligence provided by the present invention considers the dynamic changes of the network security environment and the mutual influence between homologous intelligences, and can more accurately evaluate the quality of network threat intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the accompanying drawings:

[0049] Figure 1 is a framework diagram of the comprehensive dynamic evaluation method for network threat intelligence provided by the present invention;

[0050] Figure 2 is a relationship diagram of threat intelligence sources constructed in an embodiment of the present invention;

[0051] Figure 3 is a graph showing the influence of the parameter time decay factor D on the evaluation accuracy rate, mean value of quality scores, and variance in an embodiment of the present invention;

[0052] Figure 4 is a distribution diagram of intelligence quality scores of each source in an embodiment of the present invention;

[0053] Figure 5 is a comparison diagram of IOC hit rates of different intelligence sources in an embodiment of the present invention. Specific Embodiments

[0054] To make the technical problems solved by the present invention, the technical solutions adopted, and the achieved technical effects clearer, the technical solutions of the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the invention, rather than all the embodiments. Based on the embodiments of the present invention, all other solutions obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention.

[0055] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] I. Complete Steps of the Method of the Present Invention

[0057] Step 1: Structured Definition of Cyber Threat Intelligence

[0058] CyberThreatIntelligence=(source, time, category, tag, geo, value)

[0059] Among them, CyberThreatIntelligence (CTI) represents cyber threat intelligence, source represents the intelligence source, that is, the platform where the intelligence is released, time represents the intelligence timestamp, category represents the threat type, including spam, scanner, malware, botnet, phishing, etc., tag represents the threat description label, such as IDC, attacks email, attacks mail, attacks web, attackscourierpop3, etc., geo represents the geographical location of the intelligence IP, and value represents the specific IP value or domain name.

[0060] Step 2: Evaluation of the Quality of Intelligence Sources

[0061] Construct a threat intelligence source relationship graph according to the citation relationship between intelligence sources: The threat intelligence source relationship graph is a directed graph G=(V, E, W), where each node v∈V symbolizes an intelligence source. When the intelligence that first appears in v appears in u, that is, the IP value or DNS of the intelligence i generated by the source u is the same as that of the intelligence j generated by the source v, and the timestamp of i is later than that of j, two nodes u, v∈V are connected by a directed edge (u, v)∈E, indicating that the source u quotes the intelligence of the source v. The weight of the edge (u, v) is the number of times w∈W is quoted.

[0062] The iterative algorithm for information source quality assessment evaluates the quality of information sources through two evaluation weights - content authority and link authority. The algorithm is based on two assumptions, namely, that the information generated by a high-content-authority information source will be cited by many high-link-authority information sources; and that a high-link-authority information source will cite the information generated by many high-content-authority information sources. The input of the algorithm is the information source relationship graph G. First, the content authority vector A, that is, the vector composed of the authority values of each source, and the link authority vector H, that is, the vector composed of the hub values of each source, are initialized. The lengths of both vector A and vector H are set to 1. At the k-th iteration, the authority value of source S is calculated by and the hub value of source S is calculated by . Then, vectors A and H are normalized, and the iteration is repeated until convergence. Finally, the authority values and hub values of each information source are output.

[0063] In this evaluation mechanism, more attention is paid to the originality of the source. Therefore, a source with a high authority value is considered a source with higher quality. The authority values of each source are converted to a percentage system and used as the quality score s of each piece of information i based on the information source. source ;

[0064] Step 3: Information content quality assessment

[0065] For two pieces of information with the same IP value or the same domain name, based on the four features of information source, timestamp, threat category, and description label, calculate their similarity:

[0066] S(v t , v i ) = θ1×S source + θ2×S tine + θ3×S category + θ4×S tag ;

[0067] s.t. θ1 + θ2 + θ3 + θ4 = 1;

[0068] where S(v t , v i ) represents the similarity of two threat pieces of information, S source represents the similarity of information sources, S time is the time similarity of threat information, S category is the similarity of threat categories, and S tag is the similarity of threat description labels. θ1, θ2, θ3, θ4 are the weights of these four factors respectively. In the experiment, the influences of various factors are considered equally, and the values of each weight are 0.25. For two completely identical pieces of information, their similarity is the highest, with a value of 1.

[0069] (1) Method for calculating the similarity of information sources and threat types: When the sources of two threat intelligence are the same, the similarity of information sources S source takes 1, and when they are different, S source takes 0. The similarity of threat categories S category is determined in the same way.

[0070] (2) Method for calculating time similarity:

[0071]

[0072] First, calculate the value of the time difference between the information release times |t(v t , v i ), and then normalize it to obtain the time distance Finally, calculate the time similarity S time (v t , v i ), with a value range of [0, 1]. If the timestamps t of two threat intelligence are the same, then their time distance is 0 and the time similarity S time is 1.

[0073] (3) The similarity calculation of threat description tags uses cosine similarity:

[0074]

[0075] In the formula, X t and X i are the vector representations of the threat description tags of threat intelligence v t and v i respectively, is the cosine similarity of the two vectors, with a value range of [0, 1]. When the threat intelligence tags of two threat intelligence are the same, the cosine similarity is 1. In particular, if the threat tags tag of two threat intelligence are both empty, it is stipulated that the similarity between them is 0.5.

[0076] Construct a quality evaluation index based on the intelligence content, including multi-dimensions such as content richness, content multi-source verification, timeliness, etc.:

[0077] Table 1 Summary of evaluation indicators based on intelligence content

[0078]

[0079] The preprocessed feature data is used as the input of the classifier. The KNN algorithm is applied to classify the intelligence samples into five categories according to the credibility level, and each category is assigned an intelligence quality score s content .

[0080] Step 4: Comprehensive dynamic evaluation of intelligence quality

[0081] For each threat intelligence, the quality based on the information source obtained from Step 1 and the quality based on the information content obtained from Step 2 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous threat intelligence of the same source and introducing a time decay factor to weaken the influence of information with a long time interval, its quality is evaluated:

[0082]

[0083] Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence of the same source as the nth intelligence sample; D represents the time decay factor; s i (n) represents the ith basic quality score of this intelligence sample. For example, s1(1) is the quality score of the first intelligence sample based on the information source s source , and s2(1) is the quality score of the first intelligence sample based on the information content s content ; w i is the weight of each factor s i (n), and its value needs to be adjusted according to the actual situation.

[0084] II. Method Feasibility and Beneficial Effects

[0085] Feasibility: The dataset that can be used in the implementation of the present invention is the IOC set from the threat intelligence platform Tianji Youmeng, which includes intelligence types such as feed_ipv4 and feed_domain. Each piece of intelligence has attributes such as threat category, category description, timestamp, geographical location, and credibility. Among them, the credibility score attribute can be used as a reference to verify the effectiveness of our method.

[0086] Beneficial Effects: The comprehensive quality dynamic evaluation method for network threat intelligence provided by the present invention comprehensively considers multiple indicators such as information sources and information content, and evaluates the intelligence quality from multiple dimensions; combines multiple methods such as graph mining, machine learning, and weighted average models, and fully utilizes the advantages of each technical method; considers the dynamic changes of the network security environment and the mutual influence between intelligence of the same source, and can more accurately evaluate the quality of network threat intelligence.

[0087] III. Actual Experimental Effects

[0088] Figure 2This is the threat intelligence source relationship diagram constructed in an embodiment of the present invention. The size of the nodes in the diagram is determined by the in-degree and out-degree. The larger the number of times a node points to or is pointed to, the larger the out-degree and in-degree, and the larger the size of the node. The width of the edge is determined by the weight "weight". The larger the value of "weight", the wider the edge. It can be observed that there are a total of 15 nodes and 45 edges with citation relationships among 32,142 pieces of data in the intelligence samples.

[0089] Using the intelligence source quality assessment iterative algorithm, the sample of 32,142 pieces of data converges after 118 iterations. After statistics, the average value of the Au value calculated by iteration is 0.11561544632196255, the variance is 0.053299735238440026, and the coefficient of variation is 1.9968556239783903. The coefficient of variation (coefficient of variation) is equal to the quotient of the standard deviation and the average value. The coefficient of variation is very high, close to or even exceeding 1, indicating that the degree of dispersion of the data is very large and the fluctuation range is very large. The authority values of two intelligence sources are prominent, namely Forcepoint and CoinBlockerLists, and their content authority is relatively high, that is, the intelligence generated by these two sources is cited by other intelligence sources more frequently. This result corresponds to the intuitive display of the intelligence source relationship diagram. Since this evaluation mechanism focuses on the originality of the source, such intelligence sources with high authority values are considered high-quality intelligence sources.

[0090] Figure 3 This is a graph showing the influence of the parameter time decay factor D on the evaluation accuracy rate, average quality score, and variance in an embodiment of the present invention. Experiments show that when the fixed values of the two-factor parameters are w1 = 0.001, w2 = 0.999, and D = 0.1, the matching accuracy rate is 0.9081157660392634, and as D continues to increase, the matching accuracy rate will gradually decrease; the larger D is, the smaller the variance is, that is, the smaller the fluctuation of the comprehensive quality score, which reflects that the proportion of the historical comprehensive quality score in the new comprehensive quality score is larger, and the adaptability of the algorithm to the dynamic changes of the network security environment is worse; the different values of D have no obvious influence on the average value of the comprehensive quality score. During the process of D changing from 0 to 1, the fluctuation range of the average value of the quality is within 0.02, but it can still be seen that when D takes 0.6, the average value of the quality score of the entire data set is the highest. Therefore, when using this evaluation mechanism to evaluate the threat intelligence quality of this data set, D is taken as 0.6.

[0091] Figure 4It is the distribution diagram of the intelligence quality scores of each source in an embodiment of the present invention, including the distribution range and the peak. The results show that there is indeed a gap in the intelligence quality among different intelligence sources, which verifies the necessity of our intelligence source evaluation. At the same time, it can be seen that the quality of some intelligence sources fluctuates significantly, and the score distribution range is above 40 points, such as VirusTotal, etc. This reflects that the intelligence quality generated by these sources is unstable and needs to be carefully screened. On the contrary, the intelligence quality of sources such as Openphish is relatively stable, with a fluctuation range within 20 points, and the scores are generally higher than 60 points. This indicates that the intelligence quality generated by these intelligence sources is relatively high and generally stable, so they can be trusted.

[0092] Figure 5 It is the comparison chart of the IOC hit rates of different intelligence sources in an embodiment of the present invention. It can be seen that there are obvious differences in the quality levels of the intelligence content generated by different intelligence sources, which verifies the necessity and effectiveness of the present invention in evaluating the threat intelligence quality.

[0093] IV. Embodiment

[0094] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0095] Embodiment 1:

[0096] Step 1: Intelligence source quality evaluation

[0097] According to the threat intelligence after structured representation, construct the threat intelligence source relationship diagram described in the technical solution. Through statistical analysis, it is found that there is self-citation in the threat intelligence source. Since self-citation is not helpful for judging the content authority and link authority of the original source, the self-cited edges are deleted when drawing the graph. The size of the nodes in the graph is determined by the in-degree and out-degree. The larger the number of times a node points to or is pointed to, the larger the out-degree and in-degree, and the larger the size of the node; the width of the edge is determined by the weight weight. The larger the weight value, the wider the edge. Among the 32,142 data of the intelligence samples, there are 15 nodes with citation relationships. After removing the self-citations, there are 45 edges. The threat intelligence source relationship diagram is as Figure 2 shown.

[0098] Using the iterative algorithm for evaluating the quality of information sources, the samples in this embodiment converged after 118 iterations, and the obtained authority values are shown in the table. After statistics, the average value of the authority values calculated by iteration is 0.11561544632196255, the variance is 0.053299735238440026, and the coefficient of variation is 1.9968556239783903. The coefficient of variation (coefficient of variation) is equal to the quotient of the standard deviation and the average value. The coefficient of variation is very high, close to or even exceeding 1, indicating that the degree of dispersion of the data is very large and the fluctuation range is very large. Among them, the authority values of the two information sources, Emergingthreats and GreenSnow, that is, the content authority is relatively high, indicating that the information generated by these two sources is cited by other information sources more frequently. Since this evaluation mechanism focuses on the originality of the source, such information sources with high authority values are considered high-quality information sources. The calculated authority values of each source are converted to a percentage system as the threat intelligence quality score based on the quality of the information source.

[0099] Table 2 Results of Information Source Quality Evaluation

[0100] Source Authority Emerging threats 0.725703707 GreenSnow 0.5960448799 Tor Project 0.3040962802 CoinBlockerLists 0.15.77509583 Forcepoint 0.01588434311 myip 0.01506521587 mrlooquercom 0.01228693455 VirusTotal 0.00877833709 TianJi Partners 0.00398950438 AlienVault 0.00104457209 StopForumSpam 0.00046114638 sblam 0.00039398684 Torstatus_blutmagie_de 0.00016338339 ThreatWeb 0.00008169169 Openphish 0.00002291083

[0101] Step 2: Evaluation of Information Content Quality

[0102] Implement the information similarity algorithm described in the technical solution. After statistical analysis, the average value of the similarity is 0.3475723662808495, that is, the optimal parameter value κ for judging whether other information supports this information is 0.3475723662808495.

[0103] According to the constructed feature indicators and the obtained feature vectors after quantization, after processing such as character feature numericalization and one-hot encoding, they are used as the input of the classifier. The KNN algorithm is used for multi-classification of threat intelligence samples. The classification label is the level to which the marking score score of the sample in the dataset belongs. The marking score is divided into five levels, and every 20% of the score value is a level, that is, the first 20% of the score is A, 20% - 40% is B, 40% - 60% is C, 60% - 80% is D, and the last 20% is E. In this embodiment, there are 285 marking score score values for 4941 IPV4 type intelligence samples. Every 20% is divided into a quality level, and the mode of the marking score values in each level interval will represent this level as the assignment of each level sample after classification. The score segments, score intervals, and assignments corresponding to each quality level are shown in the following table.

[0104] Table 3 Content-based Quality Level

[0105] Level Score range Score interval Assignment A Top 20% 73.8～86.0 83 B 20％～40％ 63.5～73.6 69 C 40％～60％ 53.8～63.4 58 D 60％～80％ 36.8～53.6 42 E Bottom 20% 1.0～36.4 28

[0106] After classification, each threat intelligence sample has its own quality level. The scores within each level are the same, and the score value is the mode of the values in the corresponding level interval of the marked scores in the dataset. This score is the threat intelligence quality score based on the intelligence content. Taking the score attribute marked in the dataset as the benchmark, in this embodiment, the matching accuracy rate between the content-based quality score and the marked score is 0.9230924913985024.

[0107] Step 3. Comprehensive dynamic evaluation of intelligence quality

[0108] For each threat intelligence, the quality based on the intelligence source obtained from Step 1 and the quality based on the intelligence content obtained from Step 2 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous threat intelligence of the same source and introducing a time decay factor to weaken the influence of intelligence with a long time interval, its quality is evaluated:

[0109]

[0110] Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence of the same source as the nth intelligence sample; D represents the time decay factor; s i (n) represents the ith basic quality score of this intelligence sample. For example, s1(1) is the quality score of the first intelligence sample based on the intelligence source, and s source , s2(1) is the quality score of the first intelligence sample based on the intelligence content, and s content ; w i is the weight of each factor s i (n), and its value needs to be adjusted according to the actual situation.

[0111] When conducting a static evaluation of the threat intelligence quality, that is, without considering the time decay factor D (i.e., D = 0) and only combining the two factors of intelligence source quality and intelligence content, the matching accuracy rate is the highest when the parameter values of the two factors are w1 = 0.001 and w2 = 0.999, which is 0.9253187613843351. As the proportion of the intelligence source quality factor gradually increases, the matching accuracy rate will decrease.

[0112] On this basis, considering the time dynamics, when the parameter values of the two factors are fixed as w1 = 0.001 and w2 = 0.999, when D = 0.1, the matching accuracy rate is 0.9081157660392634, and as D continues to increase, the matching accuracy rate will gradually decrease. For example, Figure 3As shown in the figure. Therefore, it can be considered that the original intelligence platform did not consider the influence of the intelligence source and the time dynamics when evaluating the quality of threat intelligence, but only evaluated the current content.

[0113] Figure 3 The figure also shows the influence curves of different values of parameter D in this embodiment on the mean and variance of the quality scores. It can be seen that the larger D is, the smaller the variance is, that is, the smaller the fluctuation of the comprehensive quality score is, which reflects that the proportion of the historical comprehensive quality score in the new comprehensive quality score is larger, and the worse the adaptability of the algorithm to the dynamic changes of the network security environment; and the influence of different values of D on the average value of the comprehensive quality score is not obvious. During the process of D changing from 0 to 1, the fluctuation range of the average value of the quality is within 0.02. However, it can still be seen that when D takes 0.6, the mean value of the quality scores of the entire data set is the highest. Therefore, D = 0.6 is taken in this embodiment.

[0114] In this embodiment, the weights of each factor are determined by the coefficient of variation method. The coefficient of variation method directly uses the information contained in each index and calculates the weights of the indexes through calculation, which is an objective method of assigning weights. Since the dimensions of the indexes in the evaluation index system are different, it is not appropriate to directly compare their differences. In order to eliminate the influence of the different dimensions of each evaluation index, it is necessary to use the coefficient of variation of each index to measure the difference degree of the values of each index. The reason why the coefficient of variation can determine the weight is that in the evaluation system, if the value of a certain factor varies greatly, it means that this factor is difficult to achieve, and it is the key factor reflecting the gap of the evaluated object, that is, a higher weight is assigned to the factor with a large value difference.

[0115] The coefficient of variation is numerically equal to the quotient of the standard deviation and the mean value, that is

[0116]

[0117] where v i is the coefficient of variation of the i-th basic quality score, σ i is the standard deviation of the i-th basic quality score, is the mean value of the i-th basic quality score.

[0118] The weight of each factor is equal to the coefficient of variation of this factor divided by the sum of the coefficients of variation of each factor, that is

[0119]

[0120] According to the above method, the weight values of the two factors of intelligence source quality and intelligence content in this embodiment are w1 = 0.83 and w2 = 0.17 respectively.

[0121] The distribution diagram of the intelligence quality scores of each source obtained in this embodiment is shown in Figure 4, including the distribution range and peak. The results show that there is indeed a gap in the intelligence quality among different intelligence sources, which verifies the necessity of our intelligence source evaluation. At the same time, it can be seen that the quality of some intelligence sources fluctuates significantly, with the score distribution range exceeding 40 points, such as VirusTotal, etc. This reflects the unstable intelligence quality generated by these sources and requires careful screening. On the contrary, the intelligence quality of sources such as Openphish is relatively stable, with the fluctuation range within 20 points, and the scores are generally higher than 60 points. This indicates that the intelligence quality generated by these intelligence sources is relatively high and generally stable, so they can be trusted.

[0122] To verify the effectiveness of the comprehensive dynamic evaluation mechanism for threat intelligence quality we proposed, we put the IOCs in the actual network environment to check the overall performance of the IOCs. Reference index Hit_IOCs_ratio: the ratio of the IOCs hit during the test to the total number of IOCs. This ratio reflects the efficiency of the IOC dataset we used to test the quality evaluation mechanism in the actual network. If a large number of IOCs evaluated by us as high-quality are hit, then to some extent, it can be considered that our evaluation mechanism is effective. We verified the IP values or domain names of the IOCs evaluated as high-quality on the VirusTotal platform and obtained a hit rate of 96.35%. For example, the IOC with the IP of 171.25.193.20 has a quality evaluation score of 82.17 under the evaluation mechanism proposed in this article and belongs to high-quality IOCs. The verification results on the VirusTotal platform show that 8 security vendors have marked it as malicious, and the public score is negative with the color marked red, then it is considered a hit. The comparison of the IOC hit rates of different intelligence sources in this embodiment can be seen in Figure 5 , it can be seen that there are obvious differences in the quality levels of the intelligence content generated by different intelligence sources, which verifies the necessity of the present invention to evaluate the threat intelligence quality. In addition, under this evaluation method, the intelligence hit rates of each source with scores greater than 80 points all exceed 85%, proving the effectiveness of the present invention.

[0123] As described above, it is only the preferred specific implementation manner of the present invention, and the protection scope of the present invention is not limited thereto. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

Claims

1. A comprehensive quality dynamic assessment method for network threat intelligence, characterized in that, Including: Step 1: Structurally define cyber threat intelligence: CyberThreatIntelligence = (source, time, category, tag, geo, value); Among them, CyberThreatIntelligence represents cyber threat intelligence, source represents the intelligence source, time represents the intelligence timestamp, category represents the threat type, tag represents the threat description label, geo represents the geographical location of the intelligence IP, and value represents the specific IP value or domain name; Step 2: Evaluate the quality of the intelligence source: Use the intelligence source quality evaluation iterative algorithm to evaluate the content authority and link authority of the intelligence source; Step 3: Evaluate the quality of the intelligence content: For two pieces of intelligence with the same IP value or domain name, calculate their similarity based on the four features of the intelligence source, timestamp, threat category, and description label; Step 4: Comprehensive dynamic evaluation of intelligence quality: For each piece of threat intelligence, the quality based on the intelligence source obtained from Step 2 and the quality based on the intelligence content obtained from Step 3 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous piece of threat intelligence of the same source and introducing a time decay factor to weaken the influence of intelligence with a long time interval, the comprehensive quality score of the nth intelligence sample is obtained; The specific content of Step 2 includes: The input of the intelligence source quality evaluation iterative algorithm is the intelligence source relationship graph G, G = (V, E, W), where each node v ∈ V represents an intelligence source; when the IP value or DNS of the intelligence i generated by the intelligence source u is the same as that of the intelligence j generated by the intelligence source v, and the timestamp of i is later than that of j, the two nodes u, v ∈ V are connected by a directed edge (u, v) ∈ E, indicating that the intelligence source u quotes the intelligence of the intelligence source v; the weight of the edge (u, v) is the number of times w ∈ W is quoted; Initialize the content authority vector A and the link authority vector H, and set the lengths of both vector A and vector H to 1; at the k-th iteration step, the authority value of the information source S is calculated by and the hub value of the information source S is calculated by ; normalize vector A and vector H, and perform iterative loops until convergence; finally, output the authority values of each information source, and convert the authority values of each information source into a percentage system as the quality score s of each intelligence i based on the information source source ; w i represents the weight of the corresponding edge, and n represents the number of connected nodes.

2. The network threat intelligence comprehensive quality dynamic evaluation method according to claim 1, wherein The specific content of Step 3 includes: S(v t ,v i ) = θ1×S source + θ2×S time + θ3×S category + θ4×S tag ; s.t. θ1 + θ2 + θ3 + θ4 = 1; s.t. means subject to; Among them, S(v t , v i ) represents the similarity between two threat intelligence, S source represents the similarity of the information source, S time is the time similarity of threat intelligence, S category is the similarity of threat categories, S tag is the similarity of threat description labels; θ1, θ2, θ3, θ4 are the weights of these four factors respectively. In the experiment, the influences of various factors are considered equally, and the value of each weight is 0.25; for two identical pieces of intelligence, their similarity is the highest, with a value of 1.

3. The network threat intelligence comprehensive quality dynamic evaluation method according to claim 2, wherein The calculation method for the similarity of information sources and the similarity of threat types: When the sources of two threat intelligence are the same, the similarity of information sources S source is taken as 1, and when they are different, S source is taken as 0; The value of the threat category similarity S category is obtained in the same way.

4. The method for dynamically evaluating the comprehensive quality of network threat intelligence according to claim 2, wherein The calculation method of the time similarity includes: First, calculate the value of the intelligence release time difference |t(v t ,v i ), and then perform normalization to obtain the time distance Finally, calculate the time similarity S time (v t ,v i ), with a value range of [0,1]; if the timestamps t of two threat intelligence are the same, then their time distance is 0 and the time similarity S time is 1.

5. The network threat intelligence comprehensive quality dynamic evaluation method according to claim 2, wherein The similarity calculation of the threat description label includes: where X t and X i are the vector representations of the threat description tags of threat intelligence v t and v i respectively, is the cosine similarity of the two vectors, with a value range of [0, 1]; when the two threat intelligence tags are the same, the cosine similarity is 1; if the threat tags tag of both threat intelligence are empty, the similarity between them is stipulated to be 0.

5.

6. The method for dynamically evaluating the comprehensive quality of network threat intelligence according to claim 2, wherein It also includes constructing quality evaluation indicators based on intelligence content. The preprocessed feature data serves as the input to the classifier. The KNN algorithm is applied to classify intelligence samples into five categories according to their credibility levels, and each category is assigned an intelligence quality score s based on the content. content 。 7. The method for dynamically evaluating the comprehensive quality of network threat intelligence according to claim 1 or 2, characterized in that The specific content of Step 4 includes: Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence homologous to the nth intelligence sample; D represents the time decay factor; s i (n) represents the ith basic quality score of this intelligence sample; w i is the weight of each factor s i (n).

8. A comprehensive quality dynamic evaluation system for network threat intelligence, characterized in that, The system is used to run the comprehensive quality dynamic evaluation method of cyber threat intelligence described in any one of claims 1-7; Including: Intelligence source quality evaluation module: Use the intelligence source quality evaluation iterative algorithm to evaluate the content authority and link authority of the intelligence source; Intelligence content quality evaluation module: For two pieces of intelligence with the same IP value or domain name, calculate their similarity based on the four features of the intelligence source, timestamp, threat category, and description label; Comprehensive Dynamic Evaluation Module for Intelligence Quality: For each threat intelligence, the quality based on the intelligence source obtained from Step 2 and the quality based on the intelligence content obtained from Step 3 are used as the basic quality scores. By adding the weighted sum of its basic quality to the comprehensive quality score of the previous threat intelligence of the same source and introducing a time decay factor to weaken the influence of intelligence with a long time interval, the comprehensive quality score of the nth intelligence sample is obtained.

9. The network threat intelligence comprehensive quality dynamic evaluation system according to claim 8, characterized in that Intelligence Source Quality Evaluation Module: Use the intelligence source quality evaluation iterative algorithm to evaluate the content authority and link authority of the intelligence source; the input of the intelligence source quality evaluation iterative algorithm is the intelligence source relationship graph G, G=(V, E, W), where each node v∈V represents an intelligence source; when the IP value or DNS of the intelligence i generated by the intelligence source u is the same as that of the intelligence j generated by the intelligence source v, and the timestamp of i is later than that of j, two nodes u, v∈V are connected by a directed edge (u, v)∈E, indicating that the intelligence source u quotes the intelligence of the intelligence source v; the weight of the edge (u, v) is the number of times w∈W is quoted; Initialize the content authority vector A and the link authority vector H, and set the lengths of both vectors A and H to 1. At the k-th iteration, the authority value of the information source S is calculated by and the hub value of the information source S is calculated by Normalize vectors A and H, and iterate in a loop until convergence. Finally, output the authority values of each information source. After converting the authority values of each information source into a percentage system, they are used as the quality scores s of each piece of information i based on the information source source ; w i represents the weight of the corresponding edge, and n represents the number of connected nodes; Intelligence Content Quality Evaluation Module: For two pieces of intelligence with the same IP value or domain name, calculate their similarity based on four features: intelligence source, timestamp, threat category, and description label; S(v t ,v i ) = θ1×S source + θ2×S time + θ3×S category + θ4×S tag ; s.t. θ1 + θ2 + θ3 + θ4 = 1; s.t. means subject to; Among them, S(v t , v i ) represents the similarity of two threat intelligence, S source represents the similarity of information sources, S time is the time similarity of threat intelligence, S category is the similarity of threat categories, S tag is the similarity of threat description tags; θ1, θ2, θ3, θ4 are the weights of these four factors respectively. In the experiment, the influence of each factor is considered equally, and the value of each weight is 0.25; for two exactly the same pieces of intelligence, their similarity is the highest, with a value of 1; Comprehensive Dynamic Evaluation Module for Information Quality: Among them, S dynamic (n) represents the comprehensive quality score of the nth intelligence sample, and S dynamic(n-1) represents the comprehensive quality score of the previous intelligence homologous to the nth intelligence sample; D represents the time decay factor; s i (n) represents the ith basic quality score of this intelligence sample; w i is the weight of each factor s i (n).

Images