Information source quality evaluation method, electronic equipment, storage medium and program product

Abstract

The invention provides an intelligence source quality evaluation method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of test samples, wherein the test samples comprise black samples and white samples; according to the plurality of test samples, determining a false alarm rate and a missing report rate corresponding to each to-be-tested intelligence source in the at least one to-be-tested intelligence source; determining at least one first intelligence source in the at least one to-be-tested intelligence source according to the false alarm rate and the missing report rate corresponding to each to-be-tested intelligence source; in a preset period, obtaining backbone network data in real time, and using the backbone network data to perform blind testing on the at least one first intelligence source to obtain alarm information corresponding to each first intelligence source in the at least one first intelligence source; and based on the false alarm rate and the missing report rate corresponding to the at least one to-be-tested intelligence source and the alarm information corresponding to the at least one first intelligence source, evaluating the quality of each to-be-tested intelligence source. And evaluation deviation is avoided.

Description

Technical Field

[0001] This application relates to the field of cybersecurity technology, and in particular to an intelligence source quality assessment method, electronic device, storage medium, and program product. Background Technology

[0002] As cyberattack methods become more sophisticated and attack chains become more covert, enterprises are increasingly reliant on threat intelligence (TI). As a core input to a cybersecurity defense system, the quality of threat intelligence directly determines the effectiveness of that system.

[0003] However, current threat intelligence sources are diverse, including commercial intelligence vendors, open-source intelligence platforms, and enterprise-developed intelligence systems. These sources exhibit significant differences in data quality, coverage, update frequency, and contextual richness. Enterprises need to integrate multi-source threat intelligence through procurement or self-development, but lack scientific evaluation methods to quantify the quality differences between sources. This can lead to false positives or false negatives in security systems due to interference from low-quality intelligence, or even allow attackers to exploit vulnerabilities in low-quality intelligence to bypass protection.

[0004] Therefore, there is an urgent need for a method to assess the quality of various intelligence sources. Summary of the Invention

[0005] This application provides a method for assessing the quality of threat intelligence sources, an electronic device, a storage medium, and a program product to optimize threat intelligence procurement strategies, improve the accuracy of security protection systems, and reduce security risks caused by defects in the quality of threat intelligence.

[0006] Firstly, this application provides a method for assessing the quality of intelligence sources, including:

[0007] Acquire multiple test samples, including black samples and white samples;

[0008] Based on the multiple test samples, determine the false positive rate and false negative rate for each of the at least one intelligence source to be tested.

[0009] Based on the false alarm rate and false negative rate of each intelligence source to be tested, at least one first intelligence source is determined among the at least one intelligence source to be tested.

[0010] Within a preset period, backbone network data is acquired in real time, and the backbone network data is used to perform blind tests on at least one first intelligence source to obtain alarm information corresponding to each of the at least one first intelligence source.

[0011] The quality of each intelligence source under test is evaluated based on the false positive rate and false negative rate corresponding to the at least one intelligence source under test, and the alarm information corresponding to the at least one first intelligence source.

[0012] In one possible implementation, evaluating the quality of each intelligence source under test based on the false positive rate and false negative rate corresponding to the at least one intelligence source under test, and the alarm information corresponding to the at least one first intelligence source, includes:

[0013] Based on the alarm information corresponding to the at least one first intelligence source, determine the score of each first intelligence source in multiple dimensions;

[0014] The scores of each second intelligence source in the multiple dimensions are set to empty, and the second intelligence source is the intelligence source other than the at least one first intelligence source among the at least one intelligence source to be tested;

[0015] Based on the false positive rate and false negative rate of the at least one intelligence source to be tested, determine the accuracy score and coverage score of each intelligence source to be tested.

[0016] Based on the scores of each intelligence source under test in multiple dimensions, including accuracy and coverage, a total score is determined for each intelligence source under test. The total score indicates the quality of the corresponding intelligence source.

[0017] In one possible implementation, the alarm information corresponding to each first intelligence source includes the total number of IOCs, the content of each alarm, the time when the first IOC generates the alarm, and the number of second IOCs. The total number of IOCs is the number of IOCs corresponding to the alarms generated by each first intelligence source after deduplication. The first IOCs are the same IOCs in the at least one first intelligence source, and the second IOCs are unique IOCs of each first intelligence source.

[0018] The step of determining the score of each first intelligence source in multiple dimensions based on the alarm information corresponding to the at least one first intelligence source includes:

[0019] Based on the number of first alarms corresponding to each first intelligence source, determine the first integral value of each first intelligence source within the preset period, wherein the first alarm is consistent with the confirmed alarm;

[0020] Based on the content of each alarm generated by each first intelligence source within the preset period, determine the second integral value of each first intelligence source within the preset period;

[0021] The third integral value of each first intelligence source within the preset period is determined according to the order in which at least one first IOC generates an alarm.

[0022] Based on the total number of IOCs, the first score, the second score, the third score, and the number of second IOCs corresponding to the at least one first intelligence source, the score of each first intelligence source in multiple dimensions is determined.

[0023] In one possible implementation, determining the score of each first intelligence source across multiple dimensions based on the total number of IOCs, the first score, the second score, the third score, and the number of second IOCs corresponding to the at least one first intelligence source includes:

[0024] The at least one first intelligence source is sorted in descending order of total IOCs to obtain a first sequence, and a first score is determined for each first intelligence source based on the first sequence.

[0025] The at least one first intelligence source is sorted in descending order of its first score value to obtain a second sequence, and a second score value for each first intelligence source is determined based on the second sequence.

[0026] The at least one first intelligence source is sorted in descending order of the second score to obtain a third sequence, and the third score of each first intelligence source is determined according to the third sequence.

[0027] The at least one first intelligence source is sorted in descending order of the third score to obtain a fourth sequence, and the fourth score of each first intelligence source is determined according to the fourth sequence.

[0028] The at least one first intelligence source is sorted in descending order of the number of second IOCs to obtain a fifth sequence, and a fifth score is determined for each first intelligence source based on the fifth sequence.

[0029] In one possible implementation, determining the accuracy score and coverage score for each intelligence source under test based on the false positive rate and false negative rate corresponding to the at least one intelligence source under test includes:

[0030] The at least one intelligence source to be tested is sorted in order of false alarm rate from low to high to obtain a sixth sequence, and the accuracy score of each intelligence source to be tested is determined based on the sixth sequence.

[0031] The at least one intelligence source to be tested is sorted in order of low to high false negative rate to obtain a seventh sequence, and the coverage score of each intelligence source to be tested is determined based on the seventh sequence.

[0032] In one possible implementation, determining the total score for each intelligence source under test based on its scores across multiple dimensions, accuracy score, and coverage score includes:

[0033] The scores of each intelligence source under test are weighted across multiple dimensions, including accuracy and coverage, to obtain the total score for each intelligence source under test.

[0034] In one possible implementation, determining at least one first intelligence source from the at least one intelligence source to be tested based on the false positive rate and false negative rate corresponding to each intelligence source to be tested includes:

[0035] The intelligence source with a false positive rate less than or equal to a first threshold and / or a false negative rate less than or equal to a second threshold among the at least one intelligence source to be tested is determined as the first intelligence source.

[0036] In one possible implementation, determining the false positive rate and false negative rate for each of the at least one intelligence source to be tested, based on the plurality of test samples, includes:

[0037] Based on the multiple test samples, the at least one intelligence source to be tested is tested respectively to obtain the test result corresponding to each intelligence source to be tested; the test result includes the test type corresponding to the multiple test samples;

[0038] For any one of the at least one intelligence sources to be tested, the false positive rate and false negative rate corresponding to the intelligence source to be tested are determined according to the test type corresponding to the plurality of test samples and the actual type corresponding to the plurality of test samples.

[0039] Secondly, this application provides an intelligence source quality assessment device, the device comprising:

[0040] The acquisition module is used to acquire multiple test samples, including black samples and white samples;

[0041] The first determining module is used to determine the false alarm rate and false negative rate of each intelligence source to be tested in at least one intelligence source to be tested, based on the plurality of test samples.

[0042] The second determining module is used to determine at least one first intelligence source among the at least one intelligence source to be tested based on the false alarm rate and false negative rate corresponding to each intelligence source to be tested.

[0043] The blind test module is used to acquire backbone network data in real time within a preset period, and use the backbone network data to perform blind tests on the at least one first intelligence source respectively, so as to obtain alarm information corresponding to each of the at least one first intelligence source.

[0044] The evaluation module is used to evaluate the quality of each intelligence source under test based on the false alarm rate and false negative rate corresponding to the at least one intelligence source under test, and the alarm information corresponding to the at least one first intelligence source.

[0045] In one possible implementation, the evaluation module is specifically used for:

[0046] Based on the alarm information corresponding to the at least one first intelligence source, determine the score of each first intelligence source in multiple dimensions;

[0047] The scores of each second intelligence source in the multiple dimensions are set to empty, and the second intelligence source is the intelligence source other than the at least one first intelligence source among the at least one intelligence source to be tested;

[0048] Based on the false positive rate and false negative rate of the at least one intelligence source to be tested, determine the accuracy score and coverage score of each intelligence source to be tested.

[0049] Based on the scores of each intelligence source under test in multiple dimensions, including accuracy and coverage, a total score is determined for each intelligence source under test. The total score indicates the quality of the corresponding intelligence source.

[0050] In one possible implementation, the alarm information corresponding to each first intelligence source includes the total number of IOCs, the content of each alarm, the time when the first IOC generates the alarm, and the number of second IOCs. The total number of IOCs is the number of duplicate IOCs corresponding to alarms generated by each first intelligence source. The first IOCs are the same IOCs among the at least one first intelligence source, and the second IOCs are unique IOCs for each first intelligence source. The evaluation module is specifically used for:

[0051] Based on the number of first alarms corresponding to each first intelligence source, determine the first integral value of each first intelligence source within the preset period, wherein the first alarm is consistent with the confirmed alarm;

[0052] Based on the content of each alarm generated by each first intelligence source within the preset period, determine the second integral value of each first intelligence source within the preset period;

[0053] The third integral value of each first intelligence source within the preset period is determined according to the order in which at least one first IOC generates an alarm.

[0054] Based on the total number of IOCs, the first score, the second score, the third score, and the number of second IOCs corresponding to the at least one first intelligence source, the score of each first intelligence source in multiple dimensions is determined.

[0055] In one possible implementation, the evaluation module is specifically used for:

[0056] The at least one first intelligence source is sorted in descending order of total IOCs to obtain a first sequence, and a first score is determined for each first intelligence source based on the first sequence.

[0057] The at least one first intelligence source is sorted in descending order of its first score value to obtain a second sequence, and a second score value for each first intelligence source is determined based on the second sequence.

[0058] The at least one first intelligence source is sorted in descending order of the second score to obtain a third sequence, and the third score of each first intelligence source is determined according to the third sequence.

[0059] The at least one first intelligence source is sorted in descending order of the third score to obtain a fourth sequence, and the fourth score of each first intelligence source is determined according to the fourth sequence.

[0060] The at least one first intelligence source is sorted in descending order of the number of second IOCs to obtain a fifth sequence, and a fifth score is determined for each first intelligence source based on the fifth sequence.

[0061] In one possible implementation, the evaluation module is specifically used for:

[0062] The at least one intelligence source to be tested is sorted in order of false alarm rate from low to high to obtain a sixth sequence, and the accuracy score of each intelligence source to be tested is determined based on the sixth sequence.

[0063] The at least one intelligence source to be tested is sorted in order of low to high false negative rate to obtain a seventh sequence, and the coverage score of each intelligence source to be tested is determined based on the seventh sequence.

[0064] In one possible implementation, the evaluation module is specifically used for:

[0065] The scores of each intelligence source under test are weighted across multiple dimensions, including accuracy and coverage, to obtain the total score for each intelligence source under test.

[0066] In one possible implementation, the second determining module is specifically used for:

[0067] The intelligence source with a false positive rate less than or equal to a first threshold and / or a false negative rate less than or equal to a second threshold among the at least one intelligence source to be tested is determined as the first intelligence source.

[0068] In one possible implementation, the first determining module is specifically used for:

[0069] Based on the multiple test samples, the at least one intelligence source to be tested is tested respectively to obtain the test result corresponding to each intelligence source to be tested; the test result includes the test type corresponding to the multiple test samples;

[0070] For any one of the at least one intelligence sources to be tested, the false positive rate and false negative rate corresponding to the intelligence source to be tested are determined according to the test type corresponding to the plurality of test samples and the actual type corresponding to the plurality of test samples.

[0071] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0072] The memory stores computer-executed instructions;

[0073] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0074] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0075] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0076] The intelligence source quality assessment method, electronic equipment, storage medium, and program products provided in this application first determine the false positive rate and false negative rate of each intelligence source through test samples. Based on the false positive rate and false negative rate, intelligence sources that meet the accuracy and coverage requirements are selected. Then, blind testing is performed on the intelligence sources that meet the accuracy and coverage requirements using backbone network data acquired in real time, obtaining alarm information for each intelligence source. Based on the false positive rate, false negative rate, and alarm information of each intelligence source, the quality of each intelligence source is evaluated. By extracting backbone network data in real time for full-scale blind testing while ensuring accuracy and coverage, the evaluation bias caused by estimations made in localized areas by traditional methods is resolved. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] Figure 1 Flowchart of the information source quality assessment method provided in this application Figure 1 ;

[0079] Figure 2 Flowchart of the information source quality assessment method provided in this application Figure 2 ;

[0080] Figure 3 Flowchart of the information source quality assessment method provided in this application Figure 3 ;

[0081] Figure 4 A schematic diagram of the intelligence source quality assessment device provided in this application;

[0082] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0083] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0084] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0085] It should be understood that the terms "comprising" or "including" indicate the presence of the described features, steps, or operations, but do not exclude the presence, occurrence, or addition of one or more other features, steps, or operations.

[0086] In this application, terms such as “exemplary,” “in some embodiments,” “for example,” and “in other embodiments” are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as “exemplary” or “for example” in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the term “exemplary” is used to present the concept in a specific manner.

[0087] The terms "first," "second," etc., used in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They do not indicate any order or limit on the number of objects in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application. For example, the use of terms such as "first intelligence source" and "second intelligence source" is only to distinguish different intelligence sources, and does not indicate any difference in the size, priority, or importance of these two intelligence sources.

[0088] To address the problems in the background technology, this application provides an intelligence source quality assessment method. Under the premise of ensuring accuracy and coverage, it extracts backbone network data in real time for full-scale blind testing, which solves the assessment bias problem caused by traditional methods that estimate within a local range. This optimizes the threat intelligence procurement strategy, improves the accuracy of the security protection system, and reduces the security risks caused by defects in threat intelligence quality.

[0089] Threat intelligence in this application refers to evidence-based knowledge about existing or potential cyber threats. It is not merely raw data (such as a bunch of log files or alarm information), but rather "intelligence" that has been analyzed, integrated, and correlated, and has context and actionability.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 1 Flowchart of the information source quality assessment method provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0092] S101. Obtain multiple test samples, including black samples and white samples.

[0093] The execution subject of this application embodiment can be an electronic device or an intelligence source quality assessment device installed in an electronic device. The intelligence source quality assessment device can be implemented by software or by a combination of software and hardware.

[0094] Multiple test samples can be obtained from different sources. For example, some test samples can be obtained from within the enterprise, and another portion can be obtained from various intelligence sources, such as intelligence providers and open-source intelligence platforms. The test samples obtained from within the enterprise can refer to samples collected internally during routine analysis.

[0095] Black samples refer to network behavior data that has been identified as malicious, while white samples refer to network behavior data that has been identified as safe or benign. Black and white samples can be used to verify the judgment capabilities of intelligence sources.

[0096] S102. Based on multiple test samples, determine the false alarm rate and false negative rate for each of the at least one intelligence source to be tested.

[0097] The intelligence source to be tested can be the same as or different from the intelligence source that provides the test sample.

[0098] After obtaining multiple test samples, the order of the test samples can be randomly shuffled before testing.

[0099] In one possible implementation, the false positive rate and false negative rate for each of the at least one intelligence source to be tested can be determined based on multiple test samples using the following method:

[0100] Based on multiple test samples, at least one intelligence source to be tested is tested to obtain test results for each intelligence source to be tested. The test results include the test types corresponding to the multiple test samples. For any intelligence source to be tested, the false positive rate and false negative rate are determined based on the test types and actual types corresponding to the multiple test samples.

[0101] For any intelligence source to be tested, test the intelligence source based on multiple test samples. This can also be described as: query all samples of the intelligence source to be tested.

[0102] For any intelligence source to be tested, the above test may include: the intelligence source to be tested determining the test type of each test sample among multiple test samples.

[0103] The test type of the test sample can indicate whether the test sample is a black sample or a white sample.

[0104] The actual type of the test sample can indicate whether the test sample is a black sample or a white sample.

[0105] Determining the false alarm rate of the intelligence source to be tested based on the test type and the actual type corresponding to multiple test samples means that the test sample whose test type is inconsistent with the actual type is identified as the first sample. The proportion of the first sample in the multiple test samples is the false alarm rate, that is, false alarm rate = number of first samples ÷ total number of multiple test samples × 100%.

[0106] The inconsistency between the test type and the actual type can mean that the test type indicates the test sample is a white sample, while the actual type indicates the test sample is a black sample; or, the test type indicates the test sample is a black sample, while the actual type indicates the test sample is a white sample.

[0107] Determining the false alarm rate of the intelligence source to be tested based on the test type and actual type corresponding to multiple test samples means that the test sample whose actual type is indicated as a black sample but whose test type is not indicated as a black sample (e.g., the test type is indicated as a white sample) is identified as the second sample. The proportion of the second sample in the multiple test samples is the false alarm rate, that is, false alarm rate = number of second samples ÷ total number of multiple test samples × 100%.

[0108] S103. Based on the false alarm rate and false negative rate of each intelligence source to be tested, determine at least one first intelligence source among at least one intelligence source to be tested.

[0109] In one possible implementation, at least one first intelligence source can be determined from at least one intelligence source to be tested based on the false positive rate and false negative rate corresponding to each intelligence source to be tested, in the following manner:

[0110] The intelligence source with a false positive rate less than or equal to a first threshold and a false negative rate less than or equal to a second threshold among at least one intelligence source to be tested is identified as the first intelligence source.

[0111] The first threshold and the second threshold can be determined according to the actual situation. This application does not impose specific restrictions on the values ​​of the first threshold and the second threshold.

[0112] That is, the first intelligence source is at least one intelligence source that meets the accuracy and coverage requirements among the intelligence sources to be tested.

[0113] S104. Within a preset period, acquire backbone network data in real time, and use the backbone network data to perform blind tests on at least one first intelligence source to obtain alarm information corresponding to each of the at least one first intelligence source.

[0114] In each test, the backbone network data used by each primary intelligence source was the same.

[0115] Backbone network data can include network traffic (NetFlow) and Domain Name System (DNS) traffic within the backbone network.

[0116] Network traffic may include at least one of the following: source Internet Protocol (IP) address, destination IP address, source port number, destination port number, protocol type, service type, timestamp of traffic start, timestamp of traffic end, number of packets or bytes.

[0117] DNS traffic can include at least one of the following: domain name, query type, source IP address, destination IP address, and the IP address corresponding to the domain name.

[0118] Within the preset period, the backbone network data is extracted in full and in real time.

[0119] For example, the preset period can be one week, but this application does not impose specific restrictions on the preset period.

[0120] Using backbone network data to conduct blind tests on at least one primary intelligence source means using backbone network data as a test sample set.

[0121] For any one of the at least one first intelligence sources, the alarm information corresponding to the first intelligence source may include: the total number of Indicators of Compromise (IOCs), the content of each alarm, the time when the first IOC generates the alarm, and the number of second IOCs. The total number of IOCs is the number of IOCs corresponding to the alarms generated by each first intelligence source after deduplication. The first IOC is the same IOC in at least one first intelligence source, and the second IOC is the unique IOC of each first intelligence source.

[0122] Each alarm can correspond to one or more IOCs, meaning that one or more IOCs can generate one alarm. For any first intelligence source, the total number of IOCs refers to the number of IOCs that generate alarms within a preset period, after deduplicating the IOCs that generate alarms for that first intelligence source.

[0123] The first IOC being the same IOC in at least one first intelligence source can also be expressed as: the first IOC being the same IOC in all first intelligence sources. For example, suppose there are 3 first intelligence sources: intelligence source 1, intelligence source 2, and intelligence source 3. The IOCs that generate alarms from intelligence source 1 include IOC1, IOC2, and IOC3; the IOCs that generate alarms from intelligence source 2 include IOC2, IOC4, and IOC5; and the IOCs that generate alarms from intelligence source 3 include IOC2, IOC6, and IOC7. Then, IOC2 is the same IOC in all first intelligence sources.

[0124] The exclusive IOC of a primary intelligence source refers to the unique IOC of that primary intelligence source. In the example above, the exclusive IOCs of intelligence source 1 are IOC2 and IOC3, the exclusive IOCs of intelligence source 2 are IOC4 and IOC5, and the exclusive IOCs of intelligence source 3 are IOC6 and IOC7.

[0125] S105. Evaluate the quality of each intelligence source under test based on the false alarm rate and false negative rate of at least one intelligence source under test, and the alarm information of at least one first intelligence source.

[0126] At least one intelligence source other than at least one first intelligence source is referred to as a second intelligence source. It should be noted that if a second intelligence source does not have corresponding alarm information, then the alarm information corresponding to the second intelligence source can be recorded as empty.

[0127] That is, for any first intelligence source, the quality of the first intelligence source is determined based on the false alarm rate, false negative rate and alarm information corresponding to the first intelligence source; for any second intelligence source, the quality of the second intelligence source is determined based on the false alarm rate and false negative rate corresponding to the second intelligence source.

[0128] exist Figure 1 In the illustrated embodiment, while ensuring accuracy (reflected by the false positive rate) and coverage (reflected by the false negative rate), backbone network data is extracted in real time for full blind testing. This solves the evaluation bias problem caused by traditional methods that estimate within a local range, thereby optimizing the threat intelligence procurement strategy, improving the accuracy of the security protection system, and reducing the security risks caused by defects in threat intelligence quality.

[0129] exist Figure 1 Based on the illustrated embodiment, the following is combined with Figure 2 The scheme of this application is described in detail.

[0130] Figure 2 Flowchart of the information source quality assessment method provided in this application Figure 2 ,like Figure 2 As shown, the method includes:

[0131] S201. Obtain multiple test samples, including black samples and white samples.

[0132] S202. Based on multiple test samples, determine the false alarm rate and false negative rate for each of the at least one intelligence source to be tested.

[0133] S203. Based on the false alarm rate and false negative rate of each intelligence source to be tested, determine at least one first intelligence source among at least one intelligence source to be tested.

[0134] S204. Within a preset period, acquire backbone network data in real time, and use the backbone network data to perform blind tests on at least one first intelligence source to obtain alarm information corresponding to each of the at least one first intelligence source.

[0135] It should be noted that the execution process of S201 to S204 can be referred to the execution process of S101 to S104, and will not be repeated here.

[0136] S205. Based on the alarm information corresponding to at least one first intelligence source, determine the score of each first intelligence source in multiple dimensions.

[0137] For example, scores across multiple dimensions may include: scores related to the total number of IOCs, scores related to the number of alarm acknowledgments, scores related to the amount of alarm content, scores related to the time when an alarm is generated by the same IOC, and scores related to the number of second IOCs.

[0138] In one possible implementation, the score of each first intelligence source in multiple dimensions can be determined based on the alarm information corresponding to at least one first intelligence source in the following manner:

[0139] Based on the number of first alarms corresponding to each first intelligence source, determine the first integral value of each first intelligence source within a preset period, where the first alarms are consistent with the confirmed alarms; based on the content of each alarm generated by each first intelligence source within the preset period, determine the second integral value of each first intelligence source within the preset period; based on the chronological order in which at least one first IOC generates alarms, determine the third integral value of each first intelligence source within the preset period; based on the total number of IOCs corresponding to at least one first intelligence source, the first integral value, the second integral value, the third integral value, and the number of second IOCs, determine the score of each first intelligence source in multiple dimensions.

[0140] For any given first intelligence source, if the alarms generated by the first intelligence source within a preset period are consistent with the confirmed alarms, then the first intelligence source can be awarded 1 point; if N alarms generated by the first intelligence source within the preset period are consistent with N confirmed alarms, then the first intelligence source can be awarded N points. Here, confirmed alarms can refer to alarms that have been manually confirmed.

[0141] The second score value for each first intelligence source within a preset period is determined based on the content of each alarm generated by that first intelligence source within the preset period. This means determining the second score value for each first intelligence source within the preset period based on the number of context fields included in each alarm generated by that first intelligence source within the preset period. For example, for any first intelligence source, the score is calculated for each context field included in the alarms generated by that first intelligence source within the preset period. For any alarm, if all context fields included in the alarm are not empty, then the first intelligence source is awarded 1 point; if each name field or category field included in the alarm exceeds 6 characters, then the first intelligence source is awarded 2 points; if each descriptive field or tactical field included in the alarm exceeds 15 characters, then the first intelligence source is awarded 4 points.

[0142] For example, intelligence source 1 generates two alarms within a preset period: alarm 1 and alarm 2. Alarm 1 includes a name field (8 characters) and a description field (10 characters), while alarm 2 includes a category field (10 characters) and a tactical field (20 fields). Therefore, intelligence source 1 can be scored 3 points (i.e., 1+2) for alarm 1 and 7 points (i.e., 1+2+4) for alarm 2. The second score value of intelligence source 1 within the preset period is 10 points.

[0143] It should be noted that the number of fields required for each type of field and the score recorded after meeting the requirements are examples only. This application does not limit the number of fields required for each type of field or the score recorded after meeting the requirements.

[0144] Based on the order in which at least one first IOC generates an alarm, the third integral value of each first intelligence source within a preset period is determined. This can mean that for any first IOC, within the preset period, the intelligence source that generates the corresponding alarm earliest is awarded 3 points, the intelligence source that generates the corresponding alarm second earliest is awarded 2 points, and the intelligence source that generates the corresponding alarm later is awarded 1 point.

[0145] It should be noted that the scores mentioned above are merely examples, and this application does not impose any restrictions on them.

[0146] S206. Set the score of each second intelligence source in multiple dimensions to empty. The second intelligence source is an intelligence source other than at least one first intelligence source among at least one intelligence source to be tested.

[0147] For example, the score of each second intelligence source can be set to 0 in multiple dimensions.

[0148] S207. Based on the false alarm rate and false negative rate of at least one intelligence source to be tested, determine the accuracy score and coverage score of each intelligence source to be tested.

[0149] In one possible implementation, the accuracy score and coverage score of each intelligence source to be tested can be determined based on the false positive rate and false negative rate corresponding to at least one intelligence source to be tested, in the following manner:

[0150] At least one intelligence source to be tested is sorted in order of false positive rate from low to high to obtain a sixth sequence, and the accuracy score of each intelligence source to be tested is determined based on the sixth sequence; at least one intelligence source to be tested is sorted in order of false negative rate from low to high to obtain a seventh sequence, and the coverage score of each intelligence source to be tested is determined based on the seventh sequence.

[0151] For example, the accuracy score of the intelligence source at the beginning of the sixth sequence is determined to be 30. Then, the accuracy scores of subsequent intelligence sources are decreased by 1 sequentially until the score reaches 0, at which point the decrease stops. That is, the accuracy score for the intelligence source with the lowest false positive rate is 30, the second lowest is 29, the third lowest is 28, the fourth lowest is 27, and so on, until an intelligence source reaches an accuracy score of 0. The accuracy scores of all intelligence sources following this source are then recorded as 0.

[0152] For example, the coverage score of the intelligence source at the beginning of the seventh sequence is determined to be 30. Then, the coverage score of each subsequent intelligence source is decreased by 1 until it reaches 0, at which point the score stops decreasing. That is, the coverage score for the intelligence source with the lowest false negative rate is 30, the second lowest is 29, the third lowest is 28, the fourth lowest is 27, and so on, until a source with a coverage score of 0 is found. The coverage score of all intelligence sources following this source is then recorded as 0.

[0153] If multiple intelligence sources have the same false positive rate, they can be randomly arranged during sorting, or they can be arranged based on other dimensions, such as false negative rate. Alternatively, they can be ranked side by side during sorting.

[0154] If multiple intelligence sources have the same false negative rate, they can be randomly arranged during sorting, or they can be arranged based on other dimensions, such as false positive rate. Alternatively, they can be ranked side by side during sorting.

[0155] It should be noted that the maximum value and the decreasing step size mentioned above are merely examples, and this application does not impose any restrictions on them.

[0156] S208. Based on the scores of each intelligence source to be tested in multiple dimensions, accuracy score and coverage score, determine the total score of each intelligence source to be tested. The total score indicates the quality of the corresponding intelligence source.

[0157] In one possible implementation, the scores of each intelligence source to be tested in multiple dimensions, including accuracy and coverage, can be weighted to obtain the total score of each intelligence source to be tested.

[0158] For example, if there are five dimensions, for any intelligence source to be tested, the scores of the first dimension, the second dimension, the third dimension, the fourth dimension, the fifth dimension, the accuracy score, and the coverage score can be weighted and calculated to obtain the total score for each intelligence source to be tested. It should be noted that the weight corresponding to each score can be determined according to the actual situation, and this application does not impose any restrictions on this.

[0159] For any intelligence source to be tested, the total score of that intelligence source is calculated as follows: (W1 × score of the first dimension + W2 × score of the second dimension + W3 × score of the third dimension + W4 × score of the fourth dimension + W5 × score of the fifth dimension + W6 × accuracy score + W7 × coverage score) ÷ preset total score. The preset total score can be the sum of the maximum score, maximum accuracy score, and maximum coverage score for each dimension.

[0160] The sum of the weights corresponding to each score can be 100 or not.

[0161] exist Figure 2 Based on the illustrated embodiment, the following is combined with Figure 3 The scheme of this application is described in detail.

[0162] Figure 3 Flowchart of the information source quality assessment method provided in this application Figure 3 ,like Figure 3 As shown, the method includes:

[0163] S301. Obtain multiple test samples, including black samples and white samples.

[0164] S302. Based on multiple test samples, test at least one intelligence source to be tested, and obtain the test results corresponding to each intelligence source to be tested in the at least one intelligence source to be tested; the test results include the test types corresponding to multiple test samples.

[0165] S303. For any one of the at least one intelligence sources to be tested, determine the false positive rate and false negative rate of the intelligence source to be tested based on the test type corresponding to the multiple test samples and the actual type corresponding to the multiple test samples.

[0166] S304. Among at least one intelligence source to be tested, the intelligence source with a false positive rate less than or equal to a first threshold and a false negative rate less than or equal to a second threshold is identified as the first intelligence source.

[0167] S305. Within a preset period, acquire backbone network data in real time, and use the backbone network data to perform blind tests on at least one first intelligence source respectively, to obtain alarm information corresponding to each first intelligence source. The alarm information corresponding to each first intelligence source includes the total number of IOCs, the content of each alarm, the time when the first IOC generates the alarm, and the number of second IOCs.

[0168] S306. Based on the number of first alarms corresponding to each first intelligence source, determine the first integral value of each first intelligence source within a preset period, and the first alarm is consistent with the confirmed alarm.

[0169] S307. Based on the content of each alarm generated by each first intelligence source within a preset period, determine the second integral value of each first intelligence source within the preset period.

[0170] S308. Determine the third integral value of each first intelligence source within a preset period according to the order in which at least one first IOC generates an alarm.

[0171] It should be noted that the execution process of S301 to S208 can be referred to the execution process of S201 to S205, and will not be repeated here.

[0172] S309. Based on the total number of IOCs corresponding to at least one first intelligence source, the first score, the second score, the third score, and the number of second IOCs, determine the score of each first intelligence source in multiple dimensions.

[0173] In one possible implementation, the score of each first intelligence source across multiple dimensions can be determined as follows, based on the total number of IOCs corresponding to at least one first intelligence source, the first score, the second score, the third score, and the number of second IOCs:

[0174] At least one first intelligence source is sorted in descending order of total IOCs to obtain a first sequence, and a first score is determined for each first intelligence source based on the first sequence. At least one first intelligence source is sorted in descending order of first score to obtain a second sequence, and a second score is determined for each first intelligence source based on the second sequence. At least one first intelligence source is sorted in descending order of second score to obtain a third sequence, and a third score is determined for each first intelligence source based on the third sequence. At least one first intelligence source is sorted in descending order of third score to obtain a fourth sequence, and a fourth score is determined for each first intelligence source based on the fourth sequence. At least one first intelligence source is sorted in descending order of second IOCs to obtain a fifth sequence, and a fifth score is determined for each first intelligence source based on the fifth sequence.

[0175] For example, the first score of the intelligence source to be tested in the first sequence is determined to be 30. Then, the first score of each subsequent intelligence source to be tested is decreased by 1 until the score reaches 0, at which point the decrease stops. That is, the first score of the intelligence source with the highest total number of IOCs is 30, the first score of the intelligence source with the second highest number of IOCs is 29, the first score of the intelligence source with the third highest number of IOCs is 28, the first score of the intelligence source with the fourth highest number of IOCs is 27, and so on, until an intelligence source with a first score of 0 is found, at which point the first score of all intelligence sources following that source is recorded as 0. The first score can also be referred to as the score related to the total number of IOCs; this application does not impose any limitation on this.

[0176] For example, the second score of the intelligence source at the beginning of the second sequence is determined to be 30. Then, the second score of each subsequent intelligence source is decreased by 1 until the score reaches 0, at which point the decrease stops. That is, the second score of the intelligence source with the highest first score is 30, the second highest is 29, the third highest is 28, the fourth highest is 27, and so on, until a intelligence source reaches a second score of 0. At this point, the second score of all intelligence sources following that source is also recorded as 0. The first score can also be referred to as the score related to the alarm confirmation count; this application does not impose any restrictions on this.

[0177] For example, the third score of the first intelligence source to be tested in the third sequence is determined to be 30. Then, the third score of each subsequent intelligence source is decreased by 1 until the score reaches 0, at which point the decrease stops. That is, the third score of the intelligence source with the highest second score is 30, the third score of the second highest second score is 29, the third score of the third highest second score is 28, the fourth highest second score is 27, and so on, until an intelligence source reaches a third score of 0. The third score of all intelligence sources following this zero is then recorded as 0. The first score can also be referred to as the score related to the amount of alarm content; this application does not limit this.

[0178] For example, the fourth score of the first intelligence source to be tested in the fourth sequence is determined to be 30. Then, the fourth score of each subsequent intelligence source is decreased by 1 until the score reaches 0, at which point the decrease stops. That is, the fourth score of the intelligence source with the highest third score is 30, the fourth score of the second highest third score is 29, the fourth score of the third highest third score is 28, the fourth score of the fourth highest third score is 27, and so on, until an intelligence source reaches a fourth score of 0. At this point, the fourth score of all intelligence sources following that source is recorded as 0. The fourth score can also be referred to as the score related to the time when an alarm is generated by the same IOC; this application does not impose any restrictions on this.

[0179] For example, the fifth score of the first intelligence source in the fifth sequence is determined to be 30. Then, the fifth score of each subsequent intelligence source in the sequence is decreased by 1 until the score reaches 0, at which point the decrease stops. That is, the fifth score of the intelligence source with the highest number of second IOCs is 30, the fifth score of the second highest number of second IOCs is 29, the fifth score of the third highest number of second IOCs is 28, the fifth score of the fourth highest number of second IOCs is 27, and so on, until an intelligence source with a fifth score of 0 is found. At this point, the fifth score of all intelligence sources following that source is recorded as 0. The fifth score can also be referred to as a score related to the number of second IOCs; this application does not limit this.

[0180] If there are multiple first intelligence sources with the same number of IOCs (or the first integral value, or the second integral value, or the third integral value, or the number of second IOCs), these multiple first intelligence sources can be randomly arranged or arranged based on other dimensions when sorting. This application does not impose any restrictions on this.

[0181] Alternatively, these multiple first intelligence sources can be ranked unequally during sorting. For example, if intelligence source 1 has 20 IOCs, intelligence source 2 has 25 IOCs, intelligence source 3 has 15 IOCs, and intelligence source 4 has 25 IOCs, then the first sequence obtained by sorting according to the number of IOCs from highest to lowest would be: intelligence source 2 (intelligence source 4), intelligence source 1, intelligence source 3, that is, intelligence source 2 and intelligence source 4 are tied for first place. Further, the first score of each intelligence source can be as follows: intelligence source 2 has a first score of 30, intelligence source 4 has a first score of 30, intelligence source 1 has a first score of 29, and intelligence source 3 has a first score of 28.

[0182] It should be noted that the scores recorded above are merely examples, and this application does not limit the recorded scores.

[0183] S310. Set the score of each second intelligence source in multiple dimensions to empty. The second intelligence source is an intelligence source other than at least one first intelligence source among at least one intelligence source to be tested.

[0184] S311. Based on the false alarm rate and false negative rate of at least one intelligence source to be tested, determine the accuracy score and coverage score of each intelligence source to be tested.

[0185] S312. Based on the scores of each intelligence source to be tested in multiple dimensions, accuracy score and coverage score, determine the total score of each intelligence source to be tested. The total score indicates the quality of the corresponding intelligence source.

[0186] It should be noted that the execution process of S310 to S312 can be referred to the execution process of S206 to S208, and will not be repeated here.

[0187] For example, based on the examples in S207 and S309, for any intelligence source to be tested, the total score of the intelligence source to be tested = (10 × first score + 10 × second score + 10 × fourth score + 10 × fifth score + 20 × third score + 20 × accuracy score + 20 × coverage score) ÷ (30 × 7).

[0188] This application embodiment uses backbone network data to evaluate the quality of threat intelligence sources based on their coverage, accuracy, richness, timeliness, and uniqueness. This avoids the evaluation bias caused by traditional estimation methods, thereby optimizing the threat intelligence procurement strategy, improving the accuracy of the security protection system, and reducing the security risks caused by defects in the quality of threat intelligence.

[0189] Figure 4 A schematic diagram of the structure of the intelligence source quality assessment device provided in this application is shown below. Figure 4 As shown, the intelligence source quality assessment device 40 provided in this embodiment includes:

[0190] The acquisition module 401 is used to acquire multiple test samples, including black samples and white samples.

[0191] The first determining module 402 is used to determine the false alarm rate and false negative rate of each intelligence source to be tested in at least one intelligence source to be tested based on multiple test samples.

[0192] The second determining module 403 is used to determine at least one first intelligence source among at least one intelligence source to be tested based on the false alarm rate and false negative rate corresponding to each intelligence source to be tested.

[0193] The blind test module 404 is used to acquire backbone network data in real time within a preset period, and use the backbone network data to perform blind tests on at least one first intelligence source respectively, so as to obtain alarm information corresponding to each first intelligence source in at least one first intelligence source.

[0194] Evaluation module 405 is used to evaluate the quality of each intelligence source under test based on the false alarm rate and false negative rate corresponding to at least one intelligence source under test, and the alarm information corresponding to at least one first intelligence source.

[0195] In one possible implementation, the evaluation module 405 is specifically used for:

[0196] Based on the alarm information corresponding to at least one primary intelligence source, determine the score of each primary intelligence source in multiple dimensions;

[0197] The scores of each second intelligence source in multiple dimensions are set to empty. A second intelligence source is an intelligence source other than at least one first intelligence source among at least one intelligence source to be tested.

[0198] Based on the false positive rate and false negative rate of at least one intelligence source to be tested, determine the accuracy score and coverage score of each intelligence source to be tested.

[0199] Based on the scores of each intelligence source under test in multiple dimensions, including accuracy and coverage, a total score is determined for each intelligence source under test. The total score indicates the quality of the corresponding intelligence source.

[0200] In one possible implementation, the alarm information corresponding to each first intelligence source includes the total number of IOCs, the content of each alarm, the time when the first IOC generates the alarm, and the number of second IOCs. The total number of IOCs is the number of duplicate IOCs corresponding to alarms generated by each first intelligence source. The first IOC is the same IOC in at least one first intelligence source, and the second IOC is a unique IOC for each first intelligence source. The evaluation module 405 is specifically used for:

[0201] Based on the number of first alarms corresponding to each first intelligence source, determine the first integral value of each first intelligence source within a preset period, and the first alarm is consistent with the confirmed alarm;

[0202] Based on the content of each alarm generated by each first intelligence source within a preset period, determine the second integral value of each first intelligence source within the preset period;

[0203] Based on the order in which at least one first IOC generates an alarm, determine the third integral value of each first intelligence source within a preset period;

[0204] Based on the total number of IOCs corresponding to at least one first intelligence source, the first score, the second score, the third score, and the number of second IOCs, the score of each first intelligence source is determined in multiple dimensions.

[0205] In one possible implementation, the evaluation module 405 is specifically used for:

[0206] At least one first intelligence source is sorted in descending order of total IOCs to obtain a first sequence, and the first score of each first intelligence source is determined based on the first sequence.

[0207] At least one first intelligence source is sorted in descending order of its first score to obtain a second sequence, and a second score is determined for each first intelligence source based on the second sequence.

[0208] At least one first intelligence source is sorted in descending order of the second score to obtain a third sequence, and the third score of each first intelligence source is determined based on the third sequence.

[0209] At least one first intelligence source is sorted in descending order of the third score to obtain a fourth sequence, and the fourth score of each first intelligence source is determined based on the fourth sequence.

[0210] At least one first intelligence source is sorted in descending order of the number of second IOCs to obtain a fifth sequence, and a fifth score is determined for each first intelligence source based on the fifth sequence.

[0211] In one possible implementation, the evaluation module 405 is specifically used for:

[0212] At least one intelligence source to be tested is sorted in order of false alarm rate from low to high to obtain a sixth sequence, and the accuracy score of each intelligence source to be tested is determined based on the sixth sequence.

[0213] At least one intelligence source to be tested is sorted in order of low to high false negative rate to obtain the seventh sequence, and the coverage score of each intelligence source to be tested is determined based on the seventh sequence.

[0214] In one possible implementation, the evaluation module 405 is specifically used for:

[0215] The scores of each intelligence source under test are weighted across multiple dimensions, including accuracy and coverage, to obtain the total score for each intelligence source under test.

[0216] In one possible implementation, the second determining module 403 is specifically used for:

[0217] The first intelligence source is defined as one of the at least one intelligence sources to be tested whose false positive rate is less than or equal to a first threshold, and / or whose false negative rate is less than or equal to a second threshold.

[0218] In one possible implementation, the first determining module 402 is specifically used for:

[0219] Based on multiple test samples, at least one intelligence source to be tested is tested respectively, and the test results corresponding to each intelligence source to be tested are obtained; the test results include the test types corresponding to multiple test samples;

[0220] For any one of the at least one intelligence sources to be tested, determine the false positive rate and false negative rate of the intelligence source to be tested based on the test type corresponding to multiple test samples and the actual type corresponding to multiple test samples.

[0221] The intelligence source quality assessment device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0222] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0223] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0224] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0225] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0226] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0227] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0228] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0229] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0230] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0231] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0232] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0235] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0237] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An intelligence source quality assessment method, characterized by, The method comprises the following steps: acquiring a plurality of test samples, the plurality of test samples comprising black samples and white samples; determining, according to the plurality of test samples, a false positive rate and a false negative rate corresponding to each of at least one to-be-tested information source; determining, according to the false positive rate and the false negative rate corresponding to each of the at least one to-be-tested information source, at least one first information source from the at least one to-be-tested information source; acquiring, in real time, backbone network data within a preset period, and performing blind testing on the at least one first information source respectively by using the backbone network data to obtain alarm information corresponding to each of the at least one first information source; evaluating the quality of each of the at least one to-be-tested information source based on the false positive rate and the false negative rate corresponding to the at least one to-be-tested information source and the alarm information corresponding to the at least one first information source.

2. The method of claim 1, wherein, The step of evaluating the quality of each of the at least one to-be-tested information source based on the false positive rate and the false negative rate corresponding to the at least one to-be-tested information source and the alarm information corresponding to the at least one first information source comprises the following steps: determining a score in multiple dimensions of each of the at least one first information source according to the alarm information corresponding to the at least one first information source; determining a score in multiple dimensions of each of the at least one to-be-tested information source as zero, the at least one to-be-tested information source being the at least one first information source; determining an accuracy score and a coverage score of each of the at least one to-be-tested information source according to the false positive rate and the false negative rate corresponding to the at least one to-be-tested information source; determining a total score of each of the at least one to-be-tested information source according to the score in multiple dimensions, the accuracy score and the coverage score of each of the at least one to-be-tested information source, the total score indicating the quality of the corresponding information source.

3. The method of claim 2, wherein, The alarm information corresponding to each of the at least one first information source comprises a total threat indicator (IOC) quantity, a content of each alarm, a time of generating an alarm by a first IOC, and a second IOC quantity, the total IOC quantity being a quantity of IOCs corresponding to alarms generated by each of the at least one first information source after deduplication, the first IOC being an identical IOC in the at least one first information source, and the second IOC being an exclusive IOC of each of the at least one first information source; The step of determining the score in multiple dimensions of each of the at least one first information source according to the alarm information corresponding to the at least one first information source comprises the following steps: determining a first integral value of each of the at least one first information source within the preset period according to a quantity of first alarms corresponding to each of the at least one first information source, the first alarms being consistent with confirmed alarms; determining a second integral value of each of the at least one first information source within the preset period according to a content of each alarm generated by each of the at least one first information source within the preset period; determining a third integral value of each of the at least one first information source within the preset period according to an order of the time of generating the alarm by the at least one first IOC; determining the score in multiple dimensions of each of the at least one first information source according to the total IOC quantity, the first integral value, the second integral value, the third integral value and the second IOC quantity corresponding to the at least one first information source.

4. The method of claim 3, wherein, The determining of the score of each first intelligence source in multiple dimensions according to the total number of IOCs corresponding to the at least one first intelligence source, the first integral value, the second integral value, the third integral value, and the number of second IOCs comprises: The at least one first intelligence source is sorted according to the total number of IOCs from high to low to obtain a first sequence, and a first score of each first intelligence source is determined according to the first sequence; The at least one first intelligence source is sorted according to the first integral value from high to low to obtain a second sequence, and a second score of each first intelligence source is determined according to the second sequence; The at least one first intelligence source is sorted according to the second integral value from high to low to obtain a third sequence, and a third score of each first intelligence source is determined according to the third sequence; The at least one first intelligence source is sorted according to the third integral value from high to low to obtain a fourth sequence, and a fourth score of each first intelligence source is determined according to the fourth sequence; The at least one first intelligence source is sorted according to the number of second IOCs from high to low to obtain a fifth sequence, and a fifth score of each first intelligence source is determined according to the fifth sequence.

5. The method according to any one of claims 2-4, characterized in that, The determining of the accuracy score and the coverage score of each to-be-tested intelligence source according to the false positive rate and the false negative rate corresponding to the at least one to-be-tested intelligence source comprises: The at least one to-be-tested intelligence source is sorted according to the false positive rate from low to high to obtain a sixth sequence, and an accuracy score of each to-be-tested intelligence source is determined according to the sixth sequence; The at least one to-be-tested intelligence source is sorted according to the false negative rate from low to high to obtain a seventh sequence, and a coverage score of each to-be-tested intelligence source is determined according to the seventh sequence.

6. The method according to any one of claims 2-5, characterized in that, The determining of the total score of each to-be-tested intelligence source according to the score in multiple dimensions, the accuracy score, and the coverage score of each to-be-tested intelligence source comprises: The score in multiple dimensions, the accuracy score, and the coverage score of each to-be-tested intelligence source are weighted to obtain the total score of each to-be-tested intelligence source.

7. The method according to any one of claims 1 to 6, characterized in that, The determining of at least one first intelligence source from the at least one to-be-tested intelligence source according to the false positive rate and the false negative rate corresponding to each to-be-tested intelligence source comprises: The intelligence source corresponding to the false positive rate less than or equal to a first threshold value and the false negative rate less than or equal to a second threshold value in the at least one to-be-tested intelligence source is determined as the first intelligence source.

8. The method according to any one of claims 1 to 7, characterized in that, The determining of the false positive rate and the false negative rate corresponding to each to-be-tested intelligence source in the at least one to-be-tested intelligence source according to the plurality of test samples comprises: The at least one to-be-tested intelligence source is tested according to the plurality of test samples to obtain the test result corresponding to each to-be-tested intelligence source in the at least one to-be-tested intelligence source; the test result comprises the test type corresponding to the plurality of test samples; For any to-be-tested intelligence source in the at least one to-be-tested intelligence source, the false positive rate and the false negative rate corresponding to the to-be-tested intelligence source are determined according to the test type corresponding to the plurality of test samples and the actual type corresponding to the plurality of test samples.

9. An information source quality assessment apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire a plurality of test samples, the plurality of test samples comprising black samples and white samples; A first determination module is configured to determine, according to the plurality of test samples, a false positive rate and a false negative rate corresponding to each of at least one to-be-tested information source; A second determination module is configured to determine, according to the false positive rate and the false negative rate corresponding to each of the at least one to-be-tested information source, at least one first information source from the at least one to-be-tested information source; A blind test module is configured to acquire, in real time, backbone network data within a preset period, and perform blind test on the at least one first information source respectively by using the backbone network data, to obtain alarm information corresponding to each of the at least one first information source; An evaluation module is configured to evaluate the quality of each of the at least one to-be-tested information source based on the false positive rate and the false negative rate corresponding to each of the at least one to-be-tested information source, and the alarm information corresponding to each of the at least one first information source.

10. An electronic device, comprising: comprise: a memory, a processor; the memory stores computer-executed instructions; the processor executes the computer-executed instructions stored in the memory, so that the processor executes the method according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the method according to any one of claims 1-8.

12. A computer program product, characterised in that, comprise a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1-8.

Images