Threat intelligence platform longitudinal evaluation method and system, and medium

Abstract

The invention discloses a threat intelligence platform longitudinal evaluation method and system and a medium, and the method comprises the steps: collecting threat information from a real-time alarm, and reversely querying the threat intelligence content of each threat intelligence platform; the threat intelligence is evaluated based on the threat intelligence source and the threat intelligence content, corresponding index scores are obtained, and the corresponding index scores comprise a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content; and obtaining a final score sorting condition of the platform by using the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content. According to the method, the defects that a real data set is not used and the content of a threat intelligence database is not analyzed in a conventional method are overcome.

Description

Technical Field

[0001] The present invention relates to the field of threat intelligence technology, and in particular to a method, system and medium for longitudinal evaluation of a threat intelligence platform based on real-time alarm data. Background Art

[0002] Nowadays, in order to fully understand the rapidly evolving cyber threat landscape and defend against increasingly complex, persistent, organized and weaponized cyber attacks, more and more organizations around the world are beginning to participate in the exchange and acquisition of cyber threat intelligence. According to the definition of the well-known security company Gartner, "Threat intelligence is a kind of evidence-based knowledge, including context, mechanism, indication, meaning and executable suggestions, which is related to existing or brewing threats or hazards faced by assets and can be used to provide information support for asset-related entities to respond to or deal with threats or hazards." Therefore, organizations can obtain actionable insights into the ever-changing cyber threat landscape from cyber threat intelligence, thereby enhancing the organization's defense capabilities and taking appropriate strategic decisions against attackers.

[0003] As cyber threat intelligence has become a basic element in the field of cybersecurity and an indispensable part of the cybersecurity architecture, research on threat intelligence has emerged in an endless stream at home and abroad. However, just as where there is demand there is sharing, and where there is sharing there is evaluation, the capabilities of threat intelligence platforms and the quality of threat intelligence they provide have become more and more the focus of attention of organizations and researchers. With the continuous advancement of research, the analysis and evaluation of threat intelligence platforms has become more diversified and in-depth. The current research has analyzed paid and free threat intelligence platforms based on quantity, accuracy, timeliness and relevance, and the focus is mainly on the push of threat intelligence platforms.

[0004] However, the current evaluation methods have the following defects: 1. Lack of real data sets. The data sets used for evaluation are basically from threat reports and whitelists, lacking real and timely malicious data. 2. Only analyzing intelligence sources. The current methods are basically based on the push of threat intelligence platforms, lacking in-depth mining of threat intelligence platform databases, while there is a large amount of effective information such as judgment history in the threat intelligence database text. 3. Only analyzing specific time points. Previous work basically only analyzed snapshots of several time nodes of the threat intelligence platform, lacking long-term and continuous attention to platform evaluation, that is, lacking in-depth vertical analysis of platform evaluation results. Summary of the invention

[0005] The main purpose of the present invention is to propose a method, system and medium for longitudinal evaluation of threat intelligence platforms based on real-time alarm data, aiming to obtain a comprehensive ranking of threat intelligence platforms by analyzing the evaluation indicators based on threat intelligence sources and threat intelligence content respectively, so as to make up for the defects of previous methods that did not use real data sets and did not analyze the contents of threat intelligence databases.

[0006] To achieve the above object, the present invention provides a threat intelligence platform vertical assessment method, the method comprising the following steps: Step S100, data collection step, collects threat information from real-time alarms, and uses the collected threat information to reversely query the threat intelligence content of each threat intelligence platform; Step S200, an indicator evaluation step, in which the threat intelligence is evaluated based on the threat intelligence source and the threat intelligence content, respectively, to obtain corresponding indicator scores, wherein the corresponding indicator scores include a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content, wherein the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation, and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation, and a content richness evaluation; Step S300, a comprehensive ranking step, uses the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content to obtain the final score ranking of the platform.

[0007] A further technical solution of the present invention is that step S100 comprises: Step S101: Acquire malicious IP data for reverse query from the real-time alarm honeypot dataset. The real-time alarm honeypot dataset is real attack data collected from the honeypot. The acquired malicious IP data includes<time,src,dst,device_type,sub_device_type,attack,type,honey_type> These fields; Step S102: Use the collected malicious IP data to reversely query the threat intelligence database through the API of each threat intelligence platform.

[0008] A further technical solution of the present invention is that in step S200, the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation, and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation, and a content richness evaluation, and the step S200 includes: Step S210, hit rate evaluation, statistics the determination results of each threat intelligence platform to obtain the hit rate analysis results; Step S220, overlap rate evaluation, statistics the consistency of the determination results of each threat intelligence platform to obtain the overlap rate analysis result; Step S230, exclusive evaluation, statistics the threat intelligence detected by each threat intelligence platform separately to obtain exclusive analysis results; Step S240, stability assessment, counting the number of flips of the judgment results of each threat intelligence platform within the time window to obtain a stability analysis result; Step S250, geographic location accuracy assessment, analyzing the judgment of geographic location in the intelligence content of each threat intelligence platform to obtain the analysis result of geographic location accuracy; Step S260, timeliness assessment, analyzing the last update time and the number of updates within the time window in each threat intelligence platform to obtain timeliness analysis results; Step S270, malicious classification situation assessment, analyzing the hit rate and extensiveness of malicious classification situations in the intelligence content of each threat intelligence platform to obtain the analysis results of the malicious classification situation; Step S280, content richness assessment, analyzes the completeness and extensiveness of the attributes in the threat intelligence content of each threat intelligence platform to obtain the analysis results of the content richness.

[0009] A further technical solution of the present invention is that step S210 includes: Step S211, determining whether the threat intelligence has a malicious judgment field, if so, directly classifying the malicious judgment result, and classifying the threat intelligence without a judgment result as undetected; Step S212, determining whether the threat intelligence has a malicious score field, and if so, segmenting the malicious score and classifying the threat intelligence according to the score segments; Step S213: if there is neither the malicious determination field nor the malicious score field, classification is performed based on historical malicious activities, and if there is no historical malicious activity, classification is performed as undetected; Step S214: derive the classification of threat intelligence for each platform, consider the threat intelligence classified as malicious and suspicious as hits, and obtain the hit rate evaluation score for each platform. ,in is the number of malicious IPs detected by platform A, It is the total number of IP addresses obtained from the alarm honey dataset.

[0010] A further technical solution of the present invention is that step S220 includes: Step S221: Calculate the overlap ratio between each platform. In platform A, the overlap ratio with platform B is ,in The number of malicious IPs detected by platform A and platform B. The number of malicious IPs detected by platform A; Step S222: Calculate the average overlap rate of each platform with the remaining platforms to obtain the overlap rate evaluation score of each platform ,in is the sum of the overlap rates of all platforms except platform A with platform A, Quantity for all platforms.

[0011] A further technical solution of the present invention is that step S230 includes: Step S231: Count the number of threat intelligence detected by each platform. In platform A, the number of malicious IPs detected is ,in is the number of malicious IPs detected by platform A, The number of malicious IPs detected for all platforms except platform A; Step S232: Divide the number of malicious IPs detected by the above platform A by the total number of malicious IPs detected by platform A to obtain the exclusiveness evaluation score of each platform .

[0012] A further technical solution of the present invention is that step S240 comprises: Step S241, counting the number of reversals of the determination of each threat intelligence on each platform within 30 days; Step S242: Calculate the average value of the threat intelligence judgment flip frequency of each platform to obtain the evaluation score of the stability of each platform ,in is the sum of the threat intelligence flip times corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A, 30 from the time window.

[0013] A further technical solution of the present invention is that step S250 comprises: Step S251: standardizing the geographic location information of each platform into ISO 3166-1 alpha-2 codes; Step S252: Collect the geographical location of the IP from the databases of the geographical location providers GeoLite2 and IP2Location. For the same IP, if the geographical locations of the two are consistent, the geographical location is regarded as a reference benchmark. If they are inconsistent, the geographical location information of all platforms is combined to select the most geographical location results as a reference benchmark. Step S253: Using the geographic location extracted in step S252 as a reference, count the geographic location judgment of each platform. In platform A, the number of IPs with accurate geographic location information is recorded as ; Step S254: Divide the number of IP addresses by the total number of IP addresses to obtain the evaluation score of the geographic location accuracy of each platform. ,in It is the total number of IP addresses obtained from the alarm honey dataset.

[0014] A further technical solution of the present invention is that step S260 comprises: Step S261: standardizing the last update time of each platform; Step S262: Count the last update time when the IP is first detected in the platform, standardize the corresponding query time and calculate the difference between the two to obtain the delay of threat intelligence on each platform. In platform A, the delay of the i-th threat intelligence is recorded as ; Step S263: Calculate the average value of the threat intelligence delay time of each platform to obtain the evaluation score of the delay time of each platform ,in is the sum of the threat intelligence delay times corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S264: Count the daily update time of each IP within 30 days to obtain the update times of each threat intelligence. In platform A, the update times of the ith threat intelligence is recorded as ; Step S265: Calculate the average number of times threat intelligence is updated on each platform to obtain an evaluation score for the number of times each platform is updated. ,in The sum of the threat intelligence updates corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S266: Comprehensively consider the above delay time and update number scores to obtain the timeliness evaluation score of each platform , where 30 comes from the time window, and The unit is day.

[0015] A further technical solution of the present invention is that step S270 includes: Step S271: Count different malicious categories of threat intelligence in each platform, and summarize and integrate these categories into several major malicious categories; Step S272: collect statistics on malicious IP behaviors reported in the real-time alarm data and classify them into several major malicious categories defined in step S271; Step S273: Count whether the malicious IP behavior in the alarm data appears in the threat intelligence, and obtain the hit rate of the malicious classification of each threat intelligence; in platform A, the hit rate of the malicious classification of the i-th threat intelligence ,in The number of malicious classification hits for threat intelligence. The number of malicious categories of the IP corresponding to the threat intelligence in the alarm data; Step S274: Calculate the average malicious classification hit rate of threat intelligence on each platform to obtain the evaluation score of the malicious classification hit rate on each platform ,in It is the sum of the threat intelligence malicious classification hit rates corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S275: Count the specific classification of malicious categories in each threat intelligence to obtain the number of malicious categories in each threat intelligence; in platform A, the number of malicious categories in the ith threat intelligence is recorded as ; Step S276: Calculate the average number of malicious categories of threat intelligence on each platform to obtain the evaluation score of the extensiveness of malicious categories on each platform ,in The sum of the number of threat intelligence malicious classifications corresponding to the malicious IPs detected by all platform A. is the number of main malicious categories defined in step S271, The number of malicious IPs detected by platform A; Step S277: Comprehensively consider the above malicious classification hit rate and extensiveness scores to obtain the evaluation scores of malicious classification of each platform ,in and Represent the malicious classification hit rate and the weight of extensiveness respectively. Here we take , .

[0016] A further technical solution of the present invention is that step S280 includes: Step S281: Count the field types in the threat intelligence of each platform and classify them into necessary attributes and extended attributes, where the number of necessary attributes is recorded as , the number of extended attributes is recorded as ; Step S282: Count the necessary attributes in the threat intelligence to obtain the number of necessary attributes filled in each threat intelligence; in platform A, the number of necessary attributes of the i-th threat intelligence is recorded as ; Step S283: Calculate the filling percentage of the necessary attributes of each platform threat intelligence to obtain the evaluation score of the attribute integrity of each platform ,in The sum of the number of necessary attributes filled in threat intelligence corresponding to all malicious IPs detected by platform A. is the number of required attributes defined in step S281, The number of malicious IPs detected by platform A; Step S284: Count the extended attributes in the threat intelligence to obtain the number of extended attributes filled in each threat intelligence; in platform A, the number of extended attributes of the ith threat intelligence is recorded as ; Step S285: Calculate the filling percentage of the extended attributes of each platform threat intelligence to obtain the evaluation score of the extensiveness of each platform attribute ,in The sum of the number of threat intelligence extended attributes filled in for all malicious IPs detected by platform A. is the extension quantity defined in step S271, The number of malicious IPs detected by platform A; Step S286: Comprehensively consider the scores of the above attribute completeness and extensiveness to obtain the evaluation score of the content richness of each platform ,in and Represent the weights of attribute completeness and extensiveness respectively, and here we take , .

[0017] A further technical solution of the present invention is that step S300 comprises: Step S310: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence source to obtain a comprehensive score based on the threat intelligence source ,in , , , are the weights of the corresponding indicators respectively; Step S320: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence content to obtain a comprehensive score based on the threat intelligence content. ,in , , , are the weights of the corresponding indicators respectively; Step S330: Utilize the comprehensive score based on threat intelligence sources Combined score based on threat intelligence content Perform weighted calculation to obtain the final platform evaluation score ,in , are the weights of the corresponding indicators respectively.

[0018] To achieve the above-mentioned objectives, the present invention also proposes a threat intelligence platform vertical assessment system, the system comprising a memory, a processor, and a threat intelligence platform vertical assessment program stored on the processor, and the threat intelligence platform vertical assessment program executes the steps of the method described above when run by the processor.

[0019] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, which stores a threat intelligence platform vertical assessment program, and the threat intelligence platform vertical assessment program executes the steps of the method described above when run by a processor.

[0020] The beneficial effects of the threat intelligence platform vertical assessment method, system and medium of the present invention are: The present invention adopts a technical solution to collect threat information from real-time alarms, and uses the collected threat information to reversely query the threat intelligence content of each threat intelligence platform; the threat intelligence is evaluated based on the threat intelligence source and the threat intelligence content, respectively, to obtain corresponding index scores, wherein the corresponding index scores include a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content, wherein the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation and a content richness evaluation; the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content are used to obtain the final score ranking of the platform, and the comprehensive ranking of the threat intelligence platform is obtained by analyzing the evaluation indicators based on the threat intelligence source and the threat intelligence content respectively, which makes up for the defects of not using real data sets and not analyzing the content of the threat intelligence database in the previous methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of a preferred embodiment of the threat intelligence platform vertical assessment method of the present invention; Figure 2is a flow chart of data collection steps provided by an embodiment of the present invention; Figure 3 is a flow chart of the indicator evaluation steps provided by an embodiment of the present invention; Figure 4 is a flow chart of the hit rate analysis steps provided by an embodiment of the present invention; Figure 5 is a flow chart of the overlap rate analysis steps provided by an embodiment of the present invention; Figure 6 is a flow chart of exclusive analysis steps provided by an embodiment of the present invention; Figure 7 is a flow chart of stability analysis steps provided by an embodiment of the present invention; Figure 8 is a flow chart of the steps of analyzing the accuracy of geographic location provided by an embodiment of the present invention; Fig. 9 is a flow chart of timeliness analysis steps provided by an embodiment of the present invention; Fig.10 is a flowchart of malicious classification situation analysis steps provided by an embodiment of the present invention; Fig.11 is a flow chart of content richness analysis steps provided by an embodiment of the present invention; Fig.12 is a flow chart of comprehensive sorting steps provided by an embodiment of the present invention; Fig.13 It is a structural block diagram of a threat intelligence platform vertical evaluation device based on real-time alarm data provided by an embodiment of the present invention.

[0023] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The present invention proposes a longitudinal evaluation method for a threat intelligence platform. The present invention obtains a comprehensive ranking of the threat intelligence platform by analyzing the evaluation indicators based on the threat intelligence source and the threat intelligence content respectively, thereby making up for the defects of previous methods that no real data set is used and no content of the threat intelligence database is analyzed.

[0026] like Figure 1As shown, a preferred embodiment of the threat intelligence platform vertical assessment method of the present invention includes the following steps: Step S100, a data collection step, collects threat information from real-time alarms, and uses the collected threat information to reversely query the threat intelligence content of each threat intelligence platform.

[0027] It should be noted that the execution subject of this embodiment can adopt the following Fig.13 The threat intelligence platform vertical evaluation device based on real-time alarm data is shown, and the threat intelligence platform vertical evaluation device based on real-time alarm data includes a data collection module 100, an indicator evaluation module 200 and a comprehensive sorting module 300. The data collection module 100 collects threat information from real-time alarms and uses the collected threat information to reversely query the threat intelligence content of each platform.

[0028] The indicator evaluation module 200 evaluates the threat intelligence based on the threat intelligence source and the threat intelligence content, and obtains corresponding indicator scores.

[0029] The comprehensive ranking module 300 uses the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content to derive the final score ranking of the platform.

[0030] See also Figure 2 The flowchart of the data collection steps shown in FIG. 1 , step S100 specifically includes the following steps: Step S101: Acquire malicious IP data for reverse query from the real-time alarm honeypot dataset. The real-time alarm honeypot dataset is real attack data collected from the honeypot. The acquired malicious IP data includes<time,src,dst,device_type,sub_device_type,attack,type,honey_type> These fields; Step S102: using the collected malicious IP data to reversely query the threat intelligence database through the API of each threat intelligence platform; Specifically, the entire reverse query process will last for 30 days, and finally obtain threat intelligence data within a 30-day time window. These data come from the databases of different threat intelligence platforms at home and abroad, such as Tencent, Qi'anxin, Virustotal, etc. In addition to the judgment of the maliciousness of the IP itself, these threat intelligence also include malicious type, last updated time, attack source, geographic location and historical detection report records.

[0031] Step S200, the indicator evaluation step, evaluates the threat intelligence based on the threat intelligence source and the threat intelligence content respectively, and obtains corresponding indicator scores, wherein the corresponding indicator scores include a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content, wherein the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation, and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation, and a content richness evaluation.

[0032] See also Figure 3 Flow chart of the indicator evaluation steps shown, in step S200, the evaluation based on the threat intelligence source includes hit rate evaluation, overlap rate evaluation, exclusivity evaluation and stability evaluation, and the evaluation based on the threat intelligence content includes geographic location accuracy evaluation, timeliness evaluation, malicious classification situation evaluation and content richness evaluation In this embodiment, based on the analysis of threat intelligence sources, these analyses are compared between platforms only based on the determination of malicious IPs on each platform, and do not involve the content attributes of the threat intelligence itself on each platform.

[0033] The step S200 specifically includes the following steps: Step S210, hit rate evaluation, statistics the judgment results of each threat intelligence platform to obtain the hit rate analysis results.

[0034] Step S220, overlap rate evaluation, statistics the consistency of the determination results of each threat intelligence platform to obtain the overlap rate analysis result.

[0035] Step S230, exclusive evaluation, statistics the threat intelligence detected by each threat intelligence platform separately to obtain exclusive analysis results.

[0036] Step S240, stability assessment, counts the number of flips of the judgment results of each threat intelligence platform within the time window to obtain a stability analysis result.

[0037] Step S250, geographic location accuracy assessment, analyzes the judgment of geographic location in the intelligence content of each threat intelligence platform to obtain the analysis result of geographic location accuracy.

[0038] Based on the analysis of the threat intelligence content, these analyses are evaluated within the platform only according to the corresponding fields in the threat intelligence, which are the content attributes of the threat intelligence of each platform. The step S200 specifically further includes the following steps: Step S260, timeliness assessment, analyzes the last update time and the number of updates within the time window in each threat intelligence platform to obtain timeliness analysis results.

[0039] Step S270, malicious classification situation evaluation, analyzing the hit rate and extensiveness of malicious classification situations in the intelligence content of each threat intelligence platform to obtain the analysis results of the malicious classification situation.

[0040] Step S280, content richness assessment, analyzes the completeness and extensiveness of the attributes in the threat intelligence content of each threat intelligence platform to obtain the analysis results of the content richness.

[0041] See also Figure 4 The flowchart of the hit rate analysis step shown in FIG. 1 , the step S210 specifically includes the following steps: Step S211, determine whether the threat intelligence has a malicious judgment field. If so, directly classify the malicious judgment result, and classify the threat intelligence without a judgment result as undetected.

[0042] For example, in the Qi'anxin platform, there is a "malicious determination" field with four classifications: "malicious", "suspicious", "non-malicious" and "unknown". The "malicious" and "suspicious" classifications are considered to have detected malicious activities, and the rest are considered to have not been detected.

[0043] Step S212: determine whether the threat intelligence has a malicious score field. If so, segment the malicious score according to the official website document and classify the threat intelligence according to the score segment.

[0044] For example, in the AbuseIPDB platform, there is an "abuseConfidenceScore" field with a score range of 0 to 100. According to the score range referenced by the official website, scores between 75 and 100 are considered malicious, 25 to 75 are considered suspicious, and the above two categories are considered to be malicious activities detected, and scores below 25 are considered to be undetected.

[0045] Step S213: If neither the malicious determination field nor the malicious score field exists, classification is performed based on historical malicious activities. If there is no historical malicious activity, classification is performed as undetected.

[0046] For example, in the AlienVault OTX platform, there is neither malicious judgment nor malicious score, so it is divided according to the number of malicious report pulses. If a certain threat intelligence has a pulse, it is considered to have detected malicious activity, and if it does not, it is considered to have not been detected.

[0047] Step S214: derive the classification of threat intelligence for each platform, consider the threat intelligence classified as malicious and suspicious as hits, and obtain the hit rate evaluation score for each platform. ,in is the number of malicious IPs detected by platform A, It is the total number of IP addresses obtained from the alarm honey dataset.

[0048] See also Figure 5 The flowchart of the overlap rate analysis step shown in FIG. 1 , the step S220 specifically includes the following steps: Step S221: Calculate the overlap ratio between each platform. In platform A, the overlap ratio with platform B is ,in The number of malicious IPs detected by platform A and platform B. The number of malicious IP addresses detected by platform A.

[0049] Step S222: Calculate the average overlap rate of each platform with the remaining platforms to obtain the overlap rate evaluation score of each platform ,in is the sum of the overlap rates of all platforms except platform A with platform A, Quantity for all platforms.

[0050] See also Figure 6 The flowchart of the exclusive analysis step shown in FIG. 1 , the S230 specifically includes the following steps: Step S231: Count the number of threat intelligence detected by each platform. In platform A, the number of malicious IPs detected is ,in is the number of malicious IPs detected by platform A, The number of malicious IPs detected for all platforms except platform A.

[0051] Step S232: Divide the number of malicious IPs detected by the above platform A by the total number of malicious IPs detected by platform A to obtain the exclusiveness evaluation score of each platform .

[0052] See also Figure 7 The flowchart of the stability analysis step shown in FIG. 1 , the S240 specifically includes the following steps: Step S241, count the number of times the judgment of each threat intelligence on each platform is flipped within 30 days.

[0053] Specifically, the "malicious" and "suspicious" judgments are regarded as 1, and the other judgment categories are regarded as 0. A string of 01 sequences with a length of 30 is obtained within the 30-day time window, where the number of flips is the number of changes from 0 to 1 and from 1 to 0 in two consecutive days. In platform A, the number of flips of the i-th threat intelligence is recorded as .

[0054] Step S242: Calculate the average value of the threat intelligence judgment flip frequency of each platform to obtain the evaluation score of the stability of each platform ,in is the sum of the threat intelligence flip times corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A, 30 from the time window.

[0055] See also Figure 8 The flowchart of the geographic location accuracy analysis step shown in FIG. 1 , the S250 specifically includes the following steps: Step S251: standardize the geographic location information of each platform and unify them into ISO 3166-1 alpha-2 codes. It should be noted that ISO 3166-1 alpha-2 codes are two-letter country codes defined by the International Organization for Standardization (ISO) and are widely used in international communication. Therefore, the geographic location information of each platform is standardized by using this code to facilitate subsequent analysis and processing, and also to be consistent with the data of the geographic location provider.

[0056] For example, in the Tencent platform, the geographic location information is in Chinese. Here, we first count the geographic locations of all countries that have appeared in threat intelligence, and then establish a dictionary that converts Chinese to ISO 3166-1 alpha-2 code to convert the geographic location information.

[0057] Step S252: Collect the geographical location of the IP from the databases of the geographical location providers GeoLite2 and IP2Location. For the same IP, if the geographical locations corresponding to the two are consistent, then the geographical location is regarded as a reference benchmark. If they are inconsistent, the geographical location information in all platforms is combined to select the most geographical location results as a reference benchmark.

[0058] Step S253: Using the geographic location extracted in the above process as a reference, count the geographic location judgment of each platform. In platform A, the number of IPs with accurate geographic location information is recorded as .

[0059] Step S254: Divide the number of IP addresses by the total number of IP addresses to obtain the evaluation score of the geographic location accuracy of each platform. ,in It is the total number of IP addresses obtained from the alarm honey dataset.

[0060] Please refer to the flowchart of the timeliness analysis steps shown in 9, the 260 specifically includes the following steps: Step S261: Standardize the last update time of each platform. For foreign platforms, directly select the corresponding time unit to convert the timestamp to UTC time. For domestic platforms, select the time zone as East 8 to convert the timestamp to UTC time.

[0061] Step S262: Count the last update time when the IP is first detected in the platform, standardize the corresponding query time and calculate the difference between the two to obtain the delay of threat intelligence on each platform. In platform A, the delay of the i-th threat intelligence is recorded as .

[0062] Step S263: Calculate the average value of the threat intelligence delay time of each platform to obtain the evaluation score of the delay time of each platform ,in is the sum of the threat intelligence delay times corresponding to all malicious IPs detected by platform A. The number of malicious IP addresses detected by platform A.

[0063] Step S264: Count the daily update time of each IP within 30 days to obtain the update times of each threat intelligence. In platform A, the update times of the ith threat intelligence is recorded as .

[0064] Step S265: Calculate the average number of times threat intelligence is updated on each platform to obtain an evaluation score for the number of times each platform is updated. ,in The sum of the threat intelligence updates corresponding to all malicious IPs detected by platform A. The number of malicious IP addresses detected by platform A.

[0065] Step S266: Comprehensively consider the above delay time and update number scores to obtain the timeliness evaluation score of each platform , where 30 comes from the time window, and The unit is day.

[0066] See also Fig.10 The flowchart of the malicious classification situation analysis step shown in FIG. 1 , the S270 specifically includes the following steps: Step S271: Count the different malicious categories of threat intelligence in each platform, and summarize and integrate these categories into several major malicious categories, such as Malware, Botnet, and Spam.

[0067] Step S272: collect statistics on the malicious IP behaviors reported in the real-time alarm data and classify them into several major malicious categories defined in step S271.

[0068] Step S273: Count whether the malicious IP behavior in the alarm data appears in the threat intelligence, and obtain the hit rate of the malicious classification of each threat intelligence; in platform A, the hit rate of the malicious classification of the i-th threat intelligence ,in The number of malicious classification hits for threat intelligence. The number of malicious categories in the alarm data corresponding to the threat intelligence IP.

[0069] Step S274: Calculate the average malicious classification hit rate of threat intelligence on each platform to obtain the evaluation score of the malicious classification hit rate on each platform ,in It is the sum of the threat intelligence malicious classification hit rates corresponding to all malicious IPs detected by platform A. The number of malicious IP addresses detected by platform A.

[0070] Step S275: Count the specific classification of malicious categories in each threat intelligence to obtain the number of malicious categories in each threat intelligence; in platform A, the number of malicious categories in the ith threat intelligence is recorded as .

[0071] Step S276: Calculate the average number of malicious categories of threat intelligence on each platform to obtain the evaluation score of the extensiveness of malicious categories on each platform ,in The sum of the number of threat intelligence malicious classifications corresponding to the malicious IPs detected by all platform A. is the number of main malicious categories defined in step S271, and is the number of malicious IPs detected by platform A.

[0072] Step S277: Comprehensively consider the above malicious classification hit rate and extensiveness scores to obtain the evaluation scores of malicious classification of each platform ,in and Represent the malicious classification hit rate and the weight of extensiveness respectively. Here we take , .

[0073] See also Fig.11 The flowchart of the content richness analysis step shown in FIG. 1 , the S280 specifically includes the following steps: Step S281: Count the field types in the threat intelligence of each platform and classify them into necessary attributes and extended attributes, where the number of necessary attributes is recorded as , the number of extended attributes is recorded as .

[0074] Step S282: Count the necessary attributes in the threat intelligence to obtain the number of necessary attributes filled in each threat intelligence; in platform A, the number of necessary attributes of the i-th threat intelligence is recorded as.

[0075] Step S283: Calculate the filling percentage of the necessary attributes of each platform threat intelligence to obtain the evaluation score of the attribute integrity of each platform ,in The sum of the number of necessary attributes filled in threat intelligence corresponding to all malicious IPs detected by platform A. is the number of required attributes defined in step S281, The number of malicious IP addresses detected by platform A.

[0076] Step S284: Count the extended attributes in the threat intelligence to obtain the number of extended attributes filled in each threat intelligence; in platform A, the number of extended attributes of the ith threat intelligence is recorded as .

[0077] Step S285: Calculate the filling percentage of the extended attributes of each platform threat intelligence to obtain the evaluation score of the extensiveness of each platform attribute ,in The sum of the number of threat intelligence extended attributes filled in for all malicious IPs detected by platform A. is the extension quantity defined in step S271, The number of malicious IP addresses detected by platform A.

[0078] Step S286: Comprehensively consider the scores of the above attribute completeness and extensiveness to obtain the evaluation score of the content richness of each platform ,in and Represent the weights of attribute completeness and extensiveness respectively, and here we take , .

[0079] Step S300, a comprehensive ranking step, uses the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content to obtain the final score ranking of the platform.

[0080] See also Fig.12 The flowchart of the comprehensive sorting steps shown in FIG. 1 , S300 specifically includes the following steps: Step S310: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence source to obtain a comprehensive score based on the threat intelligence source ,in , , , They are the weights of the corresponding indicators respectively, and security practitioners can adjust them according to their own needs.

[0081] Step S320: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence content to obtain a comprehensive score based on the threat intelligence content. ,in , , , They are the weights of the corresponding indicators respectively, and security practitioners can adjust them according to their own needs.

[0082] Step S330: Utilize the comprehensive score based on threat intelligence sources Combined score based on threat intelligence content Perform weighted calculation to obtain the final platform evaluation score ,in , They are the weights of the corresponding indicators respectively, and security practitioners can adjust them according to their own needs.

[0083] The beneficial effects of the threat intelligence platform vertical assessment method of the present invention are: The present invention adopts a technical solution to collect threat information from real-time alarms, and uses the collected threat information to reversely query the threat intelligence content of each threat intelligence platform; the threat intelligence is evaluated based on the threat intelligence source and the threat intelligence content, respectively, to obtain corresponding index scores, wherein the corresponding index scores include a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content, wherein the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation and a content richness evaluation; the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content are used to obtain the final score ranking of the platform, and the comprehensive ranking of the threat intelligence platform is obtained by analyzing the evaluation indicators based on the threat intelligence source and the threat intelligence content respectively, which makes up for the defects of not using real data sets and not analyzing the content of the threat intelligence database in the previous methods.

[0084] To achieve the above-mentioned objectives, the present invention also proposes a threat intelligence platform vertical assessment system, the system comprising a memory, a processor, and a threat intelligence platform vertical assessment program stored on the processor, the threat intelligence platform vertical assessment program executes the steps of the method described in the above embodiment when run by the processor, which will not be repeated here.

[0085] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, which stores a threat intelligence platform vertical assessment program. When the threat intelligence platform vertical assessment program is run by a processor, the steps of the method described in the above embodiment are executed, which will not be repeated here.

[0086] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A threat intelligence platform vertical assessment method, characterized in that: The method comprises the following steps: Step S100, data collection step, collects threat information from real-time alarms, and uses the collected threat information to reversely query the threat intelligence content of each threat intelligence platform; Step S200, an indicator evaluation step, in which the threat intelligence is evaluated based on the threat intelligence source and the threat intelligence content, respectively, to obtain corresponding indicator scores, wherein the corresponding indicator scores include a comprehensive score based on the threat intelligence source and a comprehensive score based on the threat intelligence content, wherein the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation, and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation, and a content richness evaluation; Step S300, a comprehensive ranking step, uses the comprehensive score based on the threat intelligence source and the comprehensive score based on the threat intelligence content to obtain the final score ranking of the platform.

2. The threat intelligence platform vertical assessment method according to claim 1 is characterized in that: The step S100 includes: Step S101: Acquire malicious IP data for reverse query from the real-time alarm honeypot dataset. The real-time alarm honeypot dataset is real attack data collected from the honeypot. The acquired malicious IP data includes<time,src,dst,device_type,sub_device_type,attack,type,honey_type> These fields; Step S102: Use the collected malicious IP data to reversely query the threat intelligence database through the API of each threat intelligence platform.

3. The threat intelligence platform vertical assessment method according to claim 2 is characterized in that: In step S200, the evaluation based on the threat intelligence source includes a hit rate evaluation, an overlap rate evaluation, an exclusivity evaluation, and a stability evaluation, and the evaluation based on the threat intelligence content includes a geographic location accuracy evaluation, a timeliness evaluation, a malicious classification situation evaluation, and a content richness evaluation. Step S200 includes: Step S210, hit rate evaluation, statistics the determination results of each threat intelligence platform to obtain the hit rate analysis results; Step S220, overlap rate evaluation, statistics the consistency of the determination results of each threat intelligence platform to obtain the overlap rate analysis result; Step S230, exclusive evaluation, statistics the threat intelligence detected by each threat intelligence platform separately to obtain exclusive analysis results; Step S240, stability assessment, counting the number of flips of the judgment results of each threat intelligence platform within the time window to obtain a stability analysis result; Step S250, geographic location accuracy assessment, analyzing the judgment of geographic location in the intelligence content of each threat intelligence platform to obtain the analysis result of geographic location accuracy; Step S260, timeliness assessment, analyzing the last update time and the number of updates within the time window in each threat intelligence platform to obtain timeliness analysis results; Step S270, malicious classification situation assessment, analyzing the hit rate and extensiveness of malicious classification situations in the intelligence content of each threat intelligence platform to obtain the analysis results of the malicious classification situation; Step S280, content richness assessment, analyzes the completeness and extensiveness of the attributes in the threat intelligence content of each threat intelligence platform to obtain the analysis results of the content richness.

4. The threat intelligence platform vertical assessment method according to claim 3 is characterized in that: The step S210 includes: Step S211, determining whether the threat intelligence has a malicious judgment field, if so, directly classifying the malicious judgment result, and classifying the threat intelligence without a judgment result as undetected; Step S212, determining whether the threat intelligence has a malicious score field, and if so, segmenting the malicious score and classifying the threat intelligence according to the score segments; Step S213: if there is neither the malicious determination field nor the malicious score field, classification is performed based on historical malicious activities, and if there is no historical malicious activity, classification is performed as undetected; Step S214: derive the classification of threat intelligence for each platform, consider the threat intelligence classified as malicious and suspicious as hits, and obtain the hit rate evaluation score for each platform. ,in is the number of malicious IPs detected by platform A, It is the total number of IP addresses obtained from the alarm honey dataset.

5. The threat intelligence platform vertical assessment method according to claim 4 is characterized in that: The step S220 includes: Step S221: Calculate the overlap ratio between each platform. In platform A, the overlap ratio with platform B is ,in The number of malicious IPs detected by platform A and platform B. The number of malicious IPs detected by platform A; Step S222: Calculate the average overlap rate of each platform with the remaining platforms to obtain the overlap rate evaluation score of each platform ,in is the sum of the overlap rates of all platforms except platform A with platform A, Quantity for all platforms.

6. The threat intelligence platform vertical assessment method according to claim 5 is characterized in that: The step S230 includes: Step S231: Count the number of threat intelligence detected by each platform. In platform A, the number of malicious IPs detected is ,in is the number of malicious IPs detected by platform A, The number of malicious IPs detected for all platforms except platform A; Step S232: Divide the number of malicious IPs detected by the above platform A by the total number of malicious IPs detected by platform A to obtain the exclusiveness evaluation score of each platform .

7. The threat intelligence platform vertical assessment method according to claim 6 is characterized in that: The step S240 includes: Step S241, counting the number of reversals of the determination of each threat intelligence on each platform within 30 days; Step S242: Calculate the average value of the threat intelligence judgment flip frequency of each platform to obtain the evaluation score of the stability of each platform ,in is the sum of the threat intelligence flip times corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A, 30 from the time window.

8. The threat intelligence platform vertical assessment method according to claim 7 is characterized in that: The step S250 includes: Step S251: standardizing the geographic location information of each platform into ISO 3166-1 alpha-2 codes; Step S252: Collect the geographical location of the IP from the databases of the geographical location providers GeoLite2 and IP2Location. For the same IP, if the geographical locations of the two are consistent, the geographical location is regarded as a reference benchmark. If they are inconsistent, the geographical location information of all platforms is combined to select the most geographical location results as a reference benchmark. Step S253: Using the geographic location extracted in step S252 as a reference, count the geographic location judgment of each platform. In platform A, the number of IPs with accurate geographic location information is recorded as ; Step S254: Divide the number of IP addresses by the total number of IP addresses to obtain the evaluation score of the geographic location accuracy of each platform. ,in It is the total number of IP addresses obtained from the alarm honey dataset.

9. The threat intelligence platform longitudinal evaluation method according to claim 8 is characterized in that: The step S260 includes: Step S261: standardizing the last update time of each platform; Step S262: Count the last update time when the IP is first detected in the platform, standardize the corresponding query time and calculate the difference between the two to obtain the delay of threat intelligence on each platform. In platform A, the delay of the i-th threat intelligence is recorded as ; Step S263: Calculate the average value of the threat intelligence delay time of each platform to obtain the evaluation score of the delay time of each platform ,in is the sum of the threat intelligence delay times corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S264: Count the daily update time of each IP within 30 days to obtain the update times of each threat intelligence. In platform A, the update times of the ith threat intelligence is recorded as ; Step S265: Calculate the average number of times threat intelligence is updated on each platform to obtain an evaluation score for the number of times each platform is updated. ,in The sum of the threat intelligence updates corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S266: Comprehensively consider the above delay time and update number scores to obtain the timeliness evaluation score of each platform , where 30 comes from the time window, and The unit is day.

10. The threat intelligence platform longitudinal evaluation method according to claim 9 is characterized in that: The step S270 includes: Step S271: Count different malicious categories of threat intelligence in each platform, and summarize and integrate these categories into several major malicious categories; Step S272: collect statistics on malicious IP behaviors reported in the real-time alarm data and classify them into several major malicious categories defined in step S271; Step S273: Count whether the malicious IP behavior in the alarm data appears in the threat intelligence, and obtain the hit rate of the malicious classification of each threat intelligence; in platform A, the hit rate of the malicious classification of the i-th threat intelligence ,in The number of malicious classification hits for threat intelligence. The number of malicious categories of the IP corresponding to the threat intelligence in the alarm data; Step S274: Calculate the average malicious classification hit rate of threat intelligence on each platform to obtain the evaluation score of the malicious classification hit rate on each platform ,in It is the sum of the threat intelligence malicious classification hit rates corresponding to all malicious IPs detected by platform A. The number of malicious IPs detected by platform A; Step S275: Count the specific classification of malicious categories in each threat intelligence to obtain the number of malicious categories in each threat intelligence; in platform A, the number of malicious categories in the ith threat intelligence is recorded as ; Step S276: Calculate the average number of malicious categories of threat intelligence on each platform to obtain the evaluation score of the extensiveness of malicious categories on each platform ,in The sum of the number of threat intelligence malicious classifications corresponding to the malicious IPs detected by all platform A. is the number of main malicious categories defined in step S271, The number of malicious IPs detected by platform A; Step S277: Comprehensively consider the above malicious classification hit rate and extensiveness scores to obtain the evaluation scores of malicious classification of each platform ,in and Represent the malicious classification hit rate and the weight of extensiveness respectively. Here we take , .

11. The threat intelligence platform vertical assessment method according to claim 10, characterized in that: The step S280 includes: Step S281: Count the field types in the threat intelligence of each platform and classify them into necessary attributes and extended attributes, where the number of necessary attributes is recorded as , the number of extended attributes is recorded as ; Step S282: Count the necessary attributes in the threat intelligence to obtain the number of necessary attributes filled in each threat intelligence; in platform A, the number of necessary attributes of the i-th threat intelligence is recorded as ; Step S283: Calculate the filling percentage of the necessary attributes of each platform threat intelligence to obtain the evaluation score of the attribute integrity of each platform ,in The sum of the number of necessary attributes filled in threat intelligence corresponding to all malicious IPs detected by platform A. is the number of required attributes defined in step S281, The number of malicious IPs detected by platform A; Step S284: Count the extended attributes in the threat intelligence to obtain the number of extended attributes filled in each threat intelligence; in platform A, the number of extended attributes of the ith threat intelligence is recorded as ; Step S285: Calculate the filling percentage of the extended attributes of each platform threat intelligence to obtain the evaluation score of the extensiveness of each platform attribute ,in The sum of the number of threat intelligence extended attributes filled in for all malicious IPs detected by platform A. is the extension quantity defined in step S271, The number of malicious IPs detected by platform A; Step S286: Comprehensively consider the scores of the above attribute completeness and extensiveness to obtain the evaluation score of the content richness of each platform ,in and Represent the weights of attribute completeness and extensiveness respectively, and here we take , .

12. The threat intelligence platform vertical assessment method according to claim 11, characterized in that: The step S300 includes: Step S310: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence source to obtain a comprehensive score based on the threat intelligence source ,in , , , are the weights of the corresponding indicators respectively; Step S320: Perform weighted calculation on the evaluation scores of the analysis results based on the threat intelligence content to obtain a comprehensive score based on the threat intelligence content. ,in , , , are the weights of the corresponding indicators respectively; Step S330: Utilize the comprehensive score based on threat intelligence sources Combined score based on threat intelligence content Perform weighted calculation to obtain the final platform evaluation score ,in , are the weights of the corresponding indicators respectively.

13. A threat intelligence platform vertical assessment system, characterized in that: The system includes a memory, a processor, and a threat intelligence platform vertical assessment program stored on the processor. When the threat intelligence platform vertical assessment program is run by the processor, the steps of the method described in any one of claims 1 to 12 are executed.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a threat intelligence platform vertical assessment program, and when the threat intelligence platform vertical assessment program is executed by a processor, the steps of the method described in any one of claims 1 to 12 are executed.

Images