A method, device and equipment for evaluating an information source

Abstract

The present application discloses a method, apparatus, and device for evaluating information sources. The method includes: obtaining information data of n information sources, and calculating the results of each information source with respect to m evaluation indicators; for each evaluation indicator, traversing the information sources and determining the importance scale value of each information source compared to all information sources when each information source is traversed. After the traversal is completed, an n×n order judgment matrix of the scheme layer is obtained; performing first eigenvector extraction and normalization on the m n×n order judgment matrices of the scheme layer to obtain m 1×n order first-level single-rank weight matrices corresponding to the evaluation indicators; according to the preset second-level single-rank weight matrix of the m evaluation indicators with respect to the information source evaluation and the 1×n order first-level single-rank weight matrix, calculating the hierarchical total-rank weight of each information source with respect to the information source evaluation and performing information source evaluation based on the hierarchical total-rank weight. The method for evaluating information sources provided by the present application provides a basis for the consumption and sharing of information data.

Description

Technical Field

[0001] This application belongs to the field of network security technology, and particularly relates to a method, device, and equipment for evaluating information sources. Background Art

[0002] With the reduction of the threshold of network attacks and the diversification of network attack means, traditional security protection devices alone can no longer meet the complex and ever-changing emerging network threats.

[0003] Currently, threat intelligence has become an important means for threat detection and security analysis. Threat intelligence abstracts the professional knowledge in network attacks, and through technical means such as security monitoring, security analysis, and security judgment, it excavates and refines data and knowledge to alleviate the asymmetry between the attacker and the defender. Threat intelligence describes the existing or upcoming threats or dangers to assets. Users can discover network threats through threat intelligence and better respond to security threats.

[0004] Information sources are the sources of threat intelligence. The prosperity of the threat intelligence market has promoted the production of threat intelligence. However, the number of information sources is numerous and the quality levels are uneven. If the quality of an information source is low, that is, some of the threat intelligence provided by the information source is not truly available threat intelligence, it will affect the detection results of threat intelligence, and thus affect the detection accuracy of network threats. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, and equipment for evaluating information sources, which are used to evaluate the quality of threat information sources and provide a basis for the consumption and sharing of threat intelligence data.

[0006] In a first aspect, this application provides a method for evaluating information sources, and the method includes:

[0007] Obtain the intelligence data of n threat information sources, and calculate the evaluation index results of each threat information source for m evaluation indicators;

[0008] For each evaluation indicator, traverse the threat information sources, and when each threat information source is traversed, use this threat information source as a benchmark. According to the importance of the benchmark threat information source compared to each threat information source and the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the benchmark threat information source compared to each threat information source. After the traversal ends, an n*n order scheme layer judgment matrix is obtained;

[0009] Perform first eigenvector extraction and normalization on the n*n order scheme layer judgment matrices corresponding to the m evaluation indicators respectively, to obtain 1*n order first-level single sorting weight matrices corresponding to the m evaluation indicators;

[0010] Calculate the hierarchical total ranking weights of each threat intelligence source for intelligence source evaluation based on the second-level single-ranking weight matrix of the preset m evaluation indicators for intelligence source evaluation and the 1*n first-level single-ranking weight matrix;

[0011] Conduct intelligence source evaluation based on the hierarchical total ranking weights of each threat intelligence source for intelligence source evaluation.

[0012] In one or more embodiments, if the intelligence data of a threat intelligence source includes k pieces of intelligence, the evaluation indicators include at least one of timeliness indicator, integrity indicator, difference indicator, credibility indicator, accuracy indicator, and harmfulness indicator, where:

[0013] Based on the timeliness weights corresponding to the time intervals between the intelligence submission times and the start times of the intelligence validity periods of the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the timeliness weights of the k pieces of intelligence and divide it by k to obtain the timeliness indicator;

[0014] Based on the filling integrity of the enriched fields of the k pieces of intelligence data reported by the threat intelligence source, calculate the value of the number of complete enriched records / the total number of reported records to obtain the integrity indicator;

[0015] Based on the duplication situation of the k pieces of intelligence data reported by the intelligence source, calculate the value of 1 - the number of duplicate records / the total number of reported records to obtain the difference indicator;

[0016] Based on the credibility values corresponding to the reputation situations of each piece of the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the credibility of the k pieces of intelligence and divide it by k to obtain the credibility indicator;

[0017] Based on the value scores of each piece of the k pieces of intelligence consumed reported by the threat intelligence source, calculate the sum of the value scores after the k pieces of intelligence are consumed and divide it by k to obtain the accuracy indicator;

[0018] Based on the harmfulness weights corresponding to the threat levels of each piece of the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the harmfulness weights of the k pieces of intelligence and divide it by k. The harmfulness weights depend on the harmfulness weights corresponding to different threat levels of the predefined intelligence to obtain the harmfulness indicator.

[0019] In one or more embodiments, according to the importance of the benchmark threat intelligence source compared with each threat intelligence source and the mapping relationship between different importance and importance scale values, obtain the importance scale values of the benchmark threat intelligence source compared with each threat intelligence source. After the traversal ends, obtain the n*n order scheme layer judgment matrix, including:

[0020] Based on the evaluation index results of m evaluation indexes from each threat intelligence source, for each evaluation index, sort the traversed benchmark threat intelligence source and each threat intelligence source;

[0021] Based on the sorting of the evaluation index results of the traversed benchmark threat intelligence source and each threat intelligence source, determine the importance of the traversed benchmark threat intelligence source compared with each threat intelligence source under this evaluation index;

[0022] Based on the mapping relationship between different importance levels and importance scale values, obtain the importance scale value corresponding to the importance of the traversed benchmark threat intelligence source compared with each threat intelligence source under this evaluation index, and the mapping relationship is the mapping relationship determined based on the Analytic Hierarchy Process (AHP);

[0023] Based on the importance scale values of each benchmark threat intelligence source compared with each threat intelligence source under this evaluation index after the traversal ends, obtain an n*n order decision-making matrix at the scheme level.

[0024] In one or more embodiments, the second-level single-sorting weight matrix of the preset m evaluation indexes for intelligence source evaluation is calculated in the following manner:

[0025] Based on the importance degree of each evaluation index for intelligence source evaluation, sort each evaluation index;

[0026] Based on the sorting result of the importance of each evaluation index for intelligence source evaluation, traverse the evaluation indexes and determine the importance of the traversed benchmark evaluation index compared with each evaluation index for intelligence source evaluation;

[0027] Based on the mapping relationship between different importance levels and importance scale values, obtain the importance scale value of the traversed benchmark evaluation index compared with each evaluation index for intelligence source evaluation, and the mapping relationship is the mapping relationship determined based on the Analytic Hierarchy Process (AHP);

[0028] Based on the importance scale values of each benchmark evaluation index compared with each evaluation index for intelligence source evaluation after the traversal ends, obtain an m*m order criterion-level judgment matrix;

[0029] Extract the second eigenvector of the m*m order criterion-level judgment matrix and normalize it to obtain a 1*m order second-level single-sorting weight matrix.

[0030] In one or more embodiments, according to the second-level single-sorting weight matrix of the preset m evaluation indexes for intelligence source evaluation and the 1*n order first-level single-sorting weight matrix, calculate the hierarchical total-sorting weight of each threat intelligence source for intelligence source evaluation, including:

[0031] According to the second-level single-rank weight matrix {a1, a2,..., a m} of the m evaluation indicators in the criterion layer with respect to the evaluation of the information source in the target layer, and the 1*n first-level single-rank weight matrix {b 1i , b 2i ,..., b ni} of the n threat information sources in the solution layer with respect to the i-th evaluation indicator in the criterion layer, calculate the 1*n hierarchical total-rank weight matrix of the i-th threat information source in the solution layer with respect to the evaluation of the information source in the target layer:

[0032]

[0033] where the value range of i is 1, 2,..., n.

[0034] In one or more embodiments, the consistency ratios CR of the solution layer judgment matrix, the criterion layer judgment matrix, and the hierarchical total-rank weight matrix all pass the consistency test, and the consistency ratio CR is calculated using the following formula:

[0035] CR = CI / RI

[0036] where CI is the consistency index, CI = (λ max - m) / (m - 1), λ max is the eigenvalue of the solution layer judgment matrix / criterion layer judgment matrix / hierarchical total-rank weight matrix, m is the number of evaluation indicators, and RI is the average random consistency index. Different m values correspond to different RI values.

[0037] In one or more embodiments, the obtaining of the intelligence data of the n threat information sources is the intelligence data of the n threat information sources obtained within a set evaluation period.

[0038] In a second aspect, the present application provides an information source evaluation device, and the device includes:

[0039] An evaluation indicator calculation module, configured to obtain the intelligence data of the n threat information sources and calculate the evaluation indicator results of each threat information source with respect to the m evaluation indicators;

[0040] A solution layer judgment matrix calculation module, configured to, for each evaluation indicator, traverse the threat information sources, and when each threat information source is traversed, use the threat information source as a benchmark, and according to the importance of the benchmark threat information source compared with each threat information source and the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the benchmark threat information source compared with each threat information source. After the traversal is completed, an n*n solution layer judgment matrix is obtained;

[0041] The scheme layer weight matrix calculation module is used to extract the first eigenvector from the n*n order scheme layer judgment matrix corresponding to each of the m evaluation indicators and normalize it to obtain the 1*n order first-level single sorting weight matrix corresponding to each of the m evaluation indicators;

[0042] The hierarchical total sorting weight calculation module is used to calculate the hierarchical total sorting weight of each threat information source for information source evaluation according to the preset second-level single sorting weight matrix of the m evaluation indicators for information source evaluation and the 1*n order first-level single sorting weight matrix;

[0043] The information source evaluation module is used to perform information source evaluation based on the hierarchical total sorting weight of each threat information source for information source evaluation.

[0044] Thirdly, an embodiment of the present application provides an information source evaluation device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the information source evaluation method provided in any one of the first aspects of the present application.

[0045] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by the processor of the terminal device, the terminal device can execute the information source evaluation method provided in any one of the first aspects of the present application.

[0046] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0047] The present application provides a method, device and equipment for information source evaluation. Based on the information data reported by each information source, taking the value characteristics of the information source as the basis, combining quantitative calculation of evaluation indicators, paying attention to various evaluation indicators and the relationships between evaluation indicators, establishing a target layer, a scheme layer, a criterion layer and corresponding hierarchical relationships, and performing information source evaluation based on the hierarchical total sorting weight of each threat information source in the scheme layer for information source evaluation in the target layer, providing a basis for the consumption and sharing of threat information data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. Obviously, the following introduced drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1Flowchart of the information source evaluation method provided by the embodiments of the present application;

[0050] Figure 2 Schematic diagram of the hierarchical structure model based on AHP provided by the embodiments of the present application;

[0051] Figure 3 Schematic diagram of the judgment matrix of the scheme layer provided by the embodiments of the present application;

[0052] Figure 4 Schematic diagram of the information source evaluation device provided by the embodiments of the present application;

[0053] Figure 5 Schematic diagram of the information source evaluation equipment provided by the embodiments of the present application. Detailed implementation manners

[0054] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Among them, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0055] To better illustrate the embodiments of the present application, the Analytic Hierarchy Process (AHP) used in the present application will be briefly introduced below. The Analytic Hierarchy Process (AHP) is a systematic and hierarchical analysis method that combines qualitative and quantitative methods. According to the nature of the problem and the overall objective to be achieved, AHP decomposes the problem into different constituent factors, and aggregates and combines the factors into different levels according to the mutual correlation and subordination relationships between the factors, forming a multi-level analysis structure model including an objective layer, a criterion layer, and a scheme layer, so as to ultimately reduce the problem to the determination of the relative importance weights or the ranking of the relative advantages and disadvantages of the lowest layer (solutions, measures, etc. for decision-making) relative to the highest layer (overall objective).

[0056] When using AHP to construct an evaluation model, it can be divided into the following four steps:

[0057] Step 1: Establish a hierarchical structure model;

[0058] Step 2: Construct a judgment (pairwise comparison) matrix;

[0059] Step 3: Hierarchical single sorting and its consistency test;

[0060] Step 4: Hierarchical total sorting and its consistency test.

[0061] Threat intelligence is evidence-based knowledge that includes context, mechanisms, indicators, implications, and practical recommendations. Threat intelligence has become an important means of threat detection and security analysis. Threat intelligence describes existing or emerging threats or risks to assets. Users can discover network threats through threat intelligence and better respond to security threats.

[0062] An information source is the source of threat intelligence. The prosperity of the threat intelligence market has promoted the production of threat intelligence. However, the number of information sources is large and the quality levels vary. If the quality of an information source is low, that is, some of the threat intelligence provided by the information source is not truly usable threat intelligence, it will affect the detection results of threat intelligence, and thus affect the detection accuracy of network threats.

[0063] In view of the above problems, the present application provides an information source evaluation method, as Figure 1 shown, the method includes:

[0064] S101, obtaining the intelligence data of n threat information sources, and calculating the evaluation index results of each threat information source for m evaluation indexes;

[0065] Based on the steps of constructing an evaluation model by AHP introduced above, in the embodiments of the present application, in order to evaluate the information source, a hierarchical structure model with the target layer being the information source evaluation, the criterion layer being m evaluation indexes, and the scheme layer being n threat information sources is constructed.

[0066] Among them, the m evaluation indexes include at least one of a timeliness index, a completeness index, a difference index, a credibility index, an accuracy index, and a harmfulness index. Exemplarily, when the criterion layer includes the above 6 evaluation indexes, as Figure 2 shown, it is a schematic diagram of the hierarchical structure model based on AHP provided by the embodiments of the present application.

[0067] As a feasible implementation manner, the obtaining of the intelligence data of n threat information sources is the intelligence data of n threat information sources obtained within a set evaluation period, and the evaluation period is set according to actual requirements. Within the set evaluation period, it is necessary to calculate the evaluation index results of each threat information source for m evaluation indexes according to the obtained intelligence data of n threat information sources.

[0068] Exemplarily, if within the current evaluation period, the intelligence data of a threat information source includes k pieces of intelligence, then the evaluation index results of each item of the threat information source are calculated in the following manner:

[0069] 1. Timeliness index

[0070] The timeliness indicator is calculated by comparing the intelligence reporting time and the start time of the intelligence validity period based on the intelligence data reported by the threat intelligence source. Exemplarily, if the time value is the absolute number of seconds starting from 00:00:00 on January 1, 1970 to the corresponding time point, then within the current evaluation period, based on the time intervals between the intelligence reporting time and the start time of the intelligence validity period of k pieces of intelligence reported by the threat intelligence source, the corresponding timeliness weights are calculated, and the sum of the timeliness weights of the k pieces of intelligence is divided by k to obtain the timeliness indicator, that is

[0071]

[0072] where k is the number of intelligence pieces, T i relates to the time interval between "intelligence reporting time" and "start time of intelligence validity period", T i is the timeliness weight value for different time intervals. Exemplarily, T i The corresponding relationship with different time intervals is shown in Table 1 below.

[0073] Table 1: Timeliness Weight Comparison Table

[0074]

[0075] 2. Integrity Indicator

[0076] The integrity is calculated based on the completeness of the enriched fields filled in the intelligence data reported by the threat intelligence source. That is, within the current evaluation period, based on the completeness of the enriched fields filled in the k pieces of intelligence data reported by the threat intelligence source, the value of the complete enriched record number / total reported record number is calculated to obtain the integrity indicator, that is, Integrity Indicator = "complete enriched record number / total reported record number" within the evaluation period.

[0077] 3. Difference Indicator

[0078] The difference is calculated based on the duplication situation reported by the intelligence source. That is, within the current evaluation period, based on the duplication situation of the k pieces of intelligence data reported by the intelligence source, the value of 1 - duplicate record number / total reported record number is calculated to obtain the difference indicator, that is

[0079] Difference Indicator = "1 - duplicate record number / total reported record number" within the evaluation period.

[0080] 4. Credibility Indicator

[0081] The credibility is calculated based on the reputation situation reported by the intelligence source. That is, within the current evaluation period, based on the credibility values corresponding to the reputation situation of each of the k pieces of intelligence reported by the threat intelligence source, the sum of the credibility of the k pieces of intelligence is divided by k to obtain the credibility indicator, that is

[0082]

[0083] Among them, R i is the value of the credibility of the intelligence, k is the number of intelligence. Exemplarily, the credibility R corresponding to the reputation of each piece of intelligence i has the values shown in Table 2 below.

[0084] Table 2: Corresponding Table of Credibility Values

[0085]

[0086] 5. Accuracy Index

[0087] The accuracy is obtained by feeding back the scoring after the consumption of threat intelligence. That is to say, in the current evaluation period, based on the value scoring of each consumed piece of intelligence among the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the value scores of the k pieces of intelligence after consumption and divide it by k to obtain the accuracy index, that is

[0088]

[0089] Among them, k is the number of consumed intelligence, and S i is the value score of a single consumed piece of intelligence. Exemplarily, S i has values from 0 to 1. If the intelligence data is consumed but not scored, the default value is 0.5.

[0090] 6. Harmfulness Index

[0091] The harmfulness is calculated based on the threat level of the threat intelligence. That is to say, in the current evaluation period, based on the harmfulness weight corresponding to the threat level of each piece of intelligence among the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the harmfulness weights of the k pieces of intelligence and divide it by k (the harmfulness weight depends on the harmfulness weights corresponding to different threat levels of the pre-defined intelligence) to obtain the harmfulness index, that is

[0092]

[0093] Among them, k is the number of intelligence, and D i is the value of the harmfulness weight corresponding to different threat levels. Exemplarily, the harmfulness weight D corresponding to different threat levels of the intelligence i has the values shown in Table 3 below.

[0094] Table 3: Corresponding Table of Harmfulness Weights

[0095]

[0096] S102. For each evaluation index, traverse the threat intelligence sources. When each threat intelligence source is traversed, use this threat intelligence source as a benchmark. According to the importance of the benchmark threat intelligence source compared to each threat intelligence source and the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the benchmark threat intelligence source compared to each threat intelligence source. After the traversal is completed, an n×n order judgment matrix of the scheme layer is obtained.

[0097] After calculating the evaluation index results of each threat intelligence source for m evaluation indexes in S101, it is necessary to perform step 2 of constructing an evaluation model using AHP: construct a judgment (pairwise comparison) matrix, that is

[0098] For each evaluation index, traverse the threat intelligence sources. When each threat intelligence source is traversed, use this threat intelligence source as a benchmark. Sort the traversed benchmark threat intelligence source and each threat intelligence source. Based on the sorting of the evaluation index results of the traversed benchmark threat intelligence source and each threat intelligence source, determine the importance of the traversed benchmark threat intelligence source compared to each threat intelligence source under this evaluation index. And based on the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values corresponding to the importance of the traversed benchmark threat intelligence source compared to each threat intelligence source under this evaluation index. The mapping relationship is a mapping relationship determined based on the Analytic Hierarchy Process (AHP), as shown in Table 4, which is the mapping relationship of importance scale values based on AHP provided by the embodiments of the present application.

[0099] Table 4: Mapping relationship of importance scale values based on AHP

[0100]

[0101] According to Table 4, the judgment matrix of all threat intelligence sources in the scheme layer regarding each index in the upper criterion layer can be obtained. If W ij = 1 and W ji = 1 in the judgment matrix under a certain evaluation index, it is considered that threat intelligence source W i and threat intelligence source W j are equally important under this evaluation index. That is to say, the importance scale values corresponding to threat intelligence source W i and threat intelligence source W j are the same under this evaluation index. Similarly, if W ij = 3 and W ji = 1 / 3 in the judgment matrix, it is considered that threat intelligence source W i is slightly more important than threat intelligence source W j under this evaluation index.

[0102] Based on the above rules, for each evaluation index, traverse each threat intelligence source, and compare the sorted results of the evaluation index results of each threat intelligence source pairwise between the benchmark threat intelligence source traversed and the evaluation index results of each threat intelligence source. Thus, the judgment matrix of the scheme layer can be obtained. After the traversal is completed, the importance scale value of each benchmark threat intelligence source compared with each threat intelligence source under each evaluation index can be obtained, and m n×n order judgment matrices of the scheme layer are obtained.

[0103] Exemplarily, when there are 6 evaluation indexes in the criterion layer, namely the timeliness index, integrity index, difference index, credibility index, accuracy index, and harmfulness index described in S101, as Figure 3 shown, it is a schematic diagram of the judgment matrix of the scheme layer provided by the embodiment of the present application.

[0104] Exemplarily, if there are 5 threat intelligence sources in the scheme layer and 6 evaluation indexes in the criterion layer, namely the 6 evaluation indexes of the timeliness index, integrity index, difference index, credibility index, accuracy index, and harmfulness index described in S101, then 6 5×5 order judgment matrices of the scheme layer are obtained after the traversal is completed.

[0105] Exemplarily, according to the sorting results of the above 5 threat intelligence sources under the integrity index, the process of obtaining the judgment matrix for the integrity index is as follows:

[0106] ① Sort the 5 threat intelligence sources according to the calculation results of the integrity index;

[0107] First place, intelligence source 1;

[0108] Second place, intelligence source 2;

[0109] Third place, intelligence source 3;

[0110] Fourth place, intelligence source 4;

[0111] Fifth place, intelligence source 5.

[0112] ② Determine the scale values in the judgment matrix according to the sorting;

[0113] Exemplarily, taking the first-place intelligence source 1 as the benchmark, according to the fact that the first place is extremely important compared with the fifth place, strongly important compared with the fourth place, significantly important compared with the third place, slightly important compared with the second place, and equally important as the first place itself, the importance scale values of the first row of the judgment matrix for the integrity index can be obtained as 1, 3, 5, 7, 9. Similarly, taking other intelligence sources as the benchmark, the importance scale values of other rows of the judgment matrix for the integrity index can be obtained, as shown in Table 5 below.

[0114] Table 5: Importance scale values for the integrity index

[0115] Integrity Index Information Source 1 Information Source 2 Information Source 3 Information Source 4 Information Source 5 Information Source 1 1 3 5 7 9 Information Source 2 1 / 3 1 3 5 7 Information Source 3 1 / 5 1 / 3 1 3 5 Information Source 4 1 / 7 1 / 5 1 / 3 1 3 Information Source 5 1 / 9 1 / 7 1 / 5 1 / 3 1

[0116] ③ Based on the importance scale value of each threat intelligence source regarding the integrity indicator, the judgment matrix of each threat intelligence source regarding the integrity indicator is constructed as follows:

[0117] W={1,3,5,7,9;1 / 3,1,3,5,7;1 / 5,1 / 3,1,3,5;1 / 7,1 / 5,1 / 3,1

[0118] , 3; 1 / 9, 1 / 7, 1 / 5, 1 / 3, 1}

[0119] It should be noted that after obtaining the above scheme-level judgment matrix, the consistency ratio CR of each judgment matrix needs to pass the consistency test to prove that the judgment matrix is reasonable and applicable. The consistency ratio CR is calculated using the following formula:

[0120] CR=CI / RI

[0121] Among them, CI is the consistency index, CI=(λ max -m) / (m-1), λ max is the eigenvalue of the solution layer judgment matrix, m is the number of evaluation indicators, and RI is the average random consistency index, as shown in Table 6. RI adopts Professor Satty’s ratio standard, and different m values correspond to different RI values.

[0122] Table 6: Average random consistency index comparison table

[0123]

[0124] The larger the CI of each judgment matrix, the more serious the inconsistency of the judgment matrix. When the consistency ratio CR is less than 0.1, it indicates that the consistency of the judgment matrix is considered to be within the allowable range. At this time, the corresponding eigenvector can be used to calculate the weight vector; if CR is greater than or equal to 0.1, the judgment matrix should be corrected.

[0125] S103, extracting and normalizing the first eigenvector of the n*n order solution layer judgment matrix corresponding to the m evaluation indicators respectively, to obtain a 1*n order first level single ranking weight matrix corresponding to the m evaluation indicators respectively;

[0126] In S102, after consistency check is performed on the n*n-order solution layer judgment matrices corresponding to the m evaluation indicators, it is determined that the m n*n-order solution layer judgment matrices are all reasonable and usable matrices.

[0127] Corresponding to step 3 of the above-mentioned method for constructing an evaluation model using AHP, that is, hierarchical single sorting and its consistency test. In S103, it is necessary to extract the first eigenvector of the m n×n order judgment matrices of the scheme layer after passing the consistency check and normalize it to obtain the 1×n order first-level single sorting weight matrices corresponding to the m evaluation indicators respectively.

[0128] The values in the 1×n order first-level single sorting weight matrices corresponding to the above-mentioned m evaluation indicators respectively represent the weights occupied by each threat intelligence source under each evaluation indicator. The larger the weight value of the threat intelligence source, the better the performance of this threat intelligence source under this evaluation indicator.

[0129] S104, according to the preset second-level single sorting weight matrix of the m evaluation indicators for the evaluation of intelligence sources, and the 1×n order first-level single sorting weight matrix, calculate the hierarchical total sorting weight of each threat intelligence source for the evaluation of intelligence sources;

[0130] S105, conduct an evaluation of intelligence sources based on the hierarchical total sorting weight of each threat intelligence source for the evaluation of intelligence sources.

[0131] In S102 and S103, it is necessary to calculate the weight values of each threat intelligence source in the scheme layer for each evaluation indicator in the upper-level criterion layer. That is, first obtain the n×n order judgment matrices of the scheme layer corresponding to the m evaluation indicators respectively. After the judgment matrices of the scheme layer all pass the consistency check, extract the first eigenvector of each judgment matrix of the scheme layer and normalize it to obtain the first-level single sorting weight matrices corresponding to the m evaluation indicators respectively;

[0132] For each evaluation indicator in the criterion layer, it is also necessary to pre-calculate the weight values of each evaluation indicator in the criterion layer for the evaluation of the intelligence source in the upper-level target layer. That is, it is necessary to pre-calculate the second-level single sorting weight matrix of the m evaluation indicators for the evaluation of intelligence sources. The calculation steps are similar to the calculation process of the first-level single sorting weight matrix in the above-mentioned S102 and S103. The specific steps are as follows:

[0133] Step 1, sort each evaluation indicator based on the importance of each evaluation indicator for the evaluation of intelligence sources;

[0134] The importance of each evaluation indicator for the evaluation of intelligence sources can be determined according to actual needs, and the importance of each evaluation indicator can be sorted.

[0135] Step 2, based on the sorting result of the importance of each evaluation indicator for the evaluation of intelligence sources, traverse the evaluation indicators, and determine the importance of the traversed reference evaluation indicator compared to each evaluation indicator for the evaluation of intelligence sources;

[0136] Step 3: Based on the mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the traversed benchmark evaluation indicators compared with each evaluation indicator for the evaluation of information sources. The mapping relationship is the one determined based on Table 4.

[0137] Step 4: Based on the importance scale values of each benchmark evaluation indicator compared with each evaluation indicator for the evaluation of information sources after the traversal, obtain an m*m order criterion layer judgment matrix.

[0138] The specific implementation manners of the above Steps 2-4 can refer to S102 and will not be elaborated here. It should be noted that the criterion layer judgment matrix also needs to pass the consistency check.

[0139] Step 5: Extract the second eigenvector of the m*m order criterion layer judgment matrix and normalize it to obtain a 1*m order second-level single sorting weight matrix.

[0140] Extract the second eigenvector of the criterion layer judgment matrix that passes the consistency check and normalize it to obtain a 1*m order second-level single sorting weight matrix {a1, a2,... a m}, where a1 represents the weight value of the first evaluation indicator among all evaluation indicators for the evaluation of information sources.

[0141] Based on the preset 1*m order second-level single sorting weight matrix {a1, a2,... a m} of the m evaluation indicators in the criterion layer for the evaluation of the target layer information source, and the 1*n order first-level single sorting weight matrix {b 1i , b 2i ,... b ni} of the n threat information sources in the solution layer for the i-th evaluation indicator in the criterion layer calculated by S103, calculate the 1*n order hierarchical total sorting weight matrix of the i-th threat information source in the solution layer for the evaluation of the target layer information source:

[0142]

[0143] Among them, the value of i is 1, 2,... n. It should be noted that the hierarchical total sorting weight matrix also needs to pass the consistency check.

[0144] The values in the 1*n order hierarchical total sorting weight matrix represent the hierarchical total sorting weight values of the n threat information sources in the solution layer for the evaluation of the target layer information source. The magnitude of the hierarchical total sorting weight value can represent the advantages and disadvantages of each threat information source. Therefore, through the hierarchical total sorting weights of each threat information source for the evaluation of the information source, the evaluation of the information source can be completed using the intelligence data reported by each threat information source.

[0145] A method for evaluating information sources provided by an embodiment of the present application takes into account the value characteristics of threat intelligence, encourages information sources to provide high-availability and complete data in a timely manner, and based on the intelligence data reported by each information source, uses the value characteristics of the information source as a basis, combines quantitative calculations of evaluation indicators, pays attention to multiple evaluation indicators and the relationships between evaluation indicators, establishes a target layer, a scheme layer, a criterion layer and corresponding hierarchical relationships, and evaluates information sources based on the total hierarchical sorting weights of each threat information source in the scheme layer regarding the evaluation of the information source in the target layer. It motivates the self-production of threat intelligence rather than mutual "copying", improves the consumption efficiency of the threat intelligence market, and promotes the high-quality and healthy development of the threat intelligence market.

[0146] Based on the same inventive concept, an embodiment of the present application also provides a device for evaluating information sources, as Figure 4 shown. The device includes:

[0147] An evaluation index calculation module 401, configured to obtain the intelligence data of n threat information sources and calculate the evaluation index results of each threat information source regarding m evaluation indicators;

[0148] A scheme layer judgment matrix calculation module 402, configured to, for each evaluation indicator, traverse the threat information sources and use each threat information source as a benchmark when traversing to a threat information source. According to the importance of the benchmark threat information source compared with each threat information source and the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the benchmark threat information source compared with each threat information source. After the traversal is completed, an n×n order scheme layer judgment matrix is obtained;

[0149] A scheme layer weight matrix calculation module 403, configured to perform first eigenvector extraction and normalization on the n×n order scheme layer judgment matrices respectively corresponding to m evaluation indicators to obtain 1×n order first-level single sorting weight matrices respectively corresponding to m evaluation indicators;

[0150] A total hierarchical sorting weight calculation module 404, configured to calculate the total hierarchical sorting weights of each threat information source regarding the evaluation of the information source according to the preset second-level single sorting weight matrix of m evaluation indicators regarding the evaluation of the information source and the 1×n order first-level single sorting weight matrix;

[0151] An information source evaluation module 405, configured to evaluate information sources based on the total hierarchical sorting weights of each threat information source regarding the evaluation of the information source.

[0152] Based on the same inventive concept, the present application also provides an information source evaluation device 500, as Figure 5As shown, it includes at least one processor 502; and a memory 501 communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method for evaluating information sources.

[0153] The memory 501 is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 501 may be a volatile memory, such as a random-access memory (RAM); it may also be a non-volatile memory, such as a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); it may also be a combination of any one or any combination of the above volatile and non-volatile memories.

[0154] The processor 502 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It may also be a hardware chip. The above hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0155] Based on the same inventive concept, an embodiment of the present application provides a computer program medium. The computer storage medium stores a computer program, and the computer program is used to cause a computer to execute the above method for evaluating information sources.

[0156] The above storage medium may be a non-temporary computer-readable storage medium. For example, the non-temporary computer-readable storage medium may be a ROM, a random-access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0158] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0159] The technical solutions provided in the present application have been introduced in detail above. Specific examples are used in the present application to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

[0160] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the present application. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be implemented on one or more computer-usable storage media containing computer-usable program code to implement each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0162] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0164] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. An information source evaluation method, characterized in that The method includes: Obtaining intelligence data from n threat intelligence sources, and calculating the evaluation index results of each threat intelligence source for m evaluation indexes; wherein, if the intelligence data of a threat intelligence source includes k pieces of intelligence, the evaluation indexes include at least two of timeliness index, integrity index, difference index, credibility index, accuracy index, and harmfulness index; wherein: based on the timeliness weight corresponding to the time interval between the intelligence reporting time and the start time of the intelligence validity period of the k pieces of intelligence reported by the threat intelligence source, calculating the sum of the timeliness weights of the k pieces of intelligence and dividing by the value of k to obtain the timeliness index; based on the filling integrity of the enriched fields of the k pieces of intelligence data reported by the threat intelligence source, calculating the value of the number of complete enriched records / total number of reported records to obtain the integrity index; based on the duplication situation of the k pieces of intelligence data reported by the intelligence source, calculating the value of 1 - number of duplicate records / total number of reported records to obtain the difference index; based on the credibility values corresponding to the reputation situation of each of the k pieces of intelligence reported by the threat intelligence source, calculating the sum of the credibility of the k pieces of intelligence and dividing by the value of k to obtain the credibility index; based on the value scores of each consumed piece of intelligence among the k pieces of intelligence reported by the threat intelligence source, calculating the sum of the value scores of the k pieces of intelligence after being consumed and dividing by the value of k to obtain the accuracy index; based on the harmfulness weights corresponding to the threat levels of each of the k pieces of intelligence reported by the threat intelligence source, calculating the sum of the harmfulness weights of the k pieces of intelligence and dividing by the value of k, and the harmfulness weight depends on the harmfulness weights corresponding to different threat levels of the predefined intelligence to obtain the harmfulness index; For each evaluation index, traverse the threat intelligence sources. When each threat intelligence source is traversed, use this threat intelligence source as the benchmark. According to the importance of the benchmark threat intelligence source compared to each threat intelligence source and the preset mapping relationship between different importance levels and importance scale values, obtain the importance scale values of the benchmark threat intelligence source compared to each threat intelligence source. After the traversal is completed, n n-order judgment matrix of the scheme layer; for the n Extract the first eigenvector of the n-order judgment matrix of the scheme layer corresponding to each of the m evaluation indexes and normalize it to obtain 1 corresponding to each of the m evaluation indexes n-order single sorting weight matrix of the first layer; According to the second-level single-rank weight matrix for the evaluation of information sources with respect to the preset m evaluation indicators, and the first-level single-rank weight matrix of the nth order, calculate the hierarchical total-rank weight of each threat information source with respect to the evaluation of information sources; Conducting intelligence source evaluation based on the total sorting weights of the hierarchical levels of the intelligence source evaluation for each threat intelligence source.

2. The method according to claim 1, characterized in that, According to the importance of the benchmark threat intelligence source compared with each threat intelligence source and the mapping relationship between different importance levels and importance scale values, the importance scale value of the benchmark threat intelligence source compared with each threat intelligence source is obtained. After the traversal ends, n is obtained. The n-order program layer judgment matrix includes: Based on the evaluation index results of each threat intelligence source for m evaluation indexes, for each evaluation index, sorting the traversed benchmark threat intelligence source and each threat intelligence source. Based on the sorting of the evaluation index results of the traversed benchmark threat intelligence source and each threat intelligence source, determining the importance of the traversed benchmark threat intelligence source compared to each threat intelligence source under this evaluation index. Based on the mapping relationship between different importance levels and importance scale values, obtaining the importance scale value corresponding to the importance of the traversed benchmark threat intelligence source compared to each threat intelligence source under this evaluation index, and the mapping relationship is the mapping relationship determined based on the Analytic Hierarchy Process (AHP). Based on the importance scale values of each benchmark threat intelligence source compared to each threat intelligence source under this evaluation index after the traversal ends, n The judgment matrix of the n-level program layer is obtained.

3. The method according to claim 1, wherein The second-level single sorting weight matrix of the preset m evaluation indexes for intelligence source evaluation is calculated in the following manner: Sorting each evaluation index based on the importance degree of each evaluation index for intelligence source evaluation. Based on the sorting result of the importance of each evaluation index for intelligence source evaluation, traversing the evaluation indexes and determining the importance of the traversed benchmark evaluation index compared to each evaluation index for intelligence source evaluation. Based on the mapping relationship between different importance levels and importance scale values, obtaining the importance scale value of the traversed benchmark evaluation index compared to each evaluation index for intelligence source evaluation, and the mapping relationship is the mapping relationship determined based on the Analytic Hierarchy Process (AHP). Based on the importance scale values of each benchmark evaluation index compared with each evaluation index for the evaluation of information sources after the traversal ends, m is obtained The judgment matrix of the criterion layer of order m; For the said m Extract the second eigenvector of the m-order criterion layer judgment matrix and normalize it to obtain 1 The m-order second-level single sorting weight matrix.

4. The method according to claim 1, characterized in that, According to the second-level single-rank weight matrix for the evaluation of information sources with respect to the preset m evaluation indicators, and the first-level single-rank weight matrix of the 1 n-order, calculate the hierarchical total-rank weight for the evaluation of each threat information source with respect to the information source evaluation, including: According to the second-level single-rank weight matrix { } of the m evaluation indicators in the criterion layer with respect to the evaluation of the information source in the target layer, and the 1 n-order first-level single-rank weight matrix { } of the n threat information sources in the solution layer with respect to the i-th evaluation indicator in the criterion layer, calculate the 1 n-order hierarchical total-rank weight matrix of the i-th threat information source in the solution layer with respect to the evaluation of the information source in the target layer: Wherein, the value of i is 1, 2,..., n.

5. The method according to claim 2, 3 or 4, characterized in that The consistency ratios of the scheme layer judgment matrix, criterion layer judgment matrix, and hierarchical total ranking weight matrix all pass the consistency test, and the consistency ratio is calculated using the following formula: Among them, is the consistency index, , is the eigenvalue of the judgment matrix at the scheme layer / criterion layer / hierarchical total sorting weight matrix, m is the number of evaluation indicators, is the average random consistency index, and different m values correspond to different values.

6. The method according to claim 1, wherein The obtaining of the intelligence data of n threat intelligence sources is the intelligence data of n threat intelligence sources obtained within a set evaluation period.

7. An information source evaluation device, characterized in that, It includes: An evaluation index calculation module, configured to obtain the intelligence data of n threat intelligence sources and calculate the evaluation index results of each threat intelligence source for m evaluation indexes; wherein, if the intelligence data of a threat intelligence source includes k pieces of intelligence, the evaluation indexes include at least two of a timeliness index, a completeness index, a difference index, a credibility index, an accuracy index, and a harmfulness index; wherein: based on the timeliness weight corresponding to the time interval between the intelligence reporting time and the start time of the intelligence validity period of the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the timeliness weights of the k pieces of intelligence and divide by the value of k to obtain the timeliness index; based on the filling completeness of the enriched fields of the k pieces of intelligence data reported by the threat intelligence source, calculate the value of the number of complete enriched records / the total number of reported records to obtain the completeness index; based on the repetition situation of the k pieces of intelligence data reported by the intelligence source, calculate the value of 1 - the number of repeated records / the total number of reported records to obtain the difference index; based on the credibility value corresponding to the credibility of each piece of intelligence among the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the credibility of the k pieces of intelligence and divide by the value of k to obtain the credibility index; based on the value scoring of each piece of consumed intelligence among the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the value scores of the k pieces of intelligence after being consumed and divide by the value of k to obtain the accuracy index; based on the harmfulness weight corresponding to the threat level of each piece of intelligence among the k pieces of intelligence reported by the threat intelligence source, calculate the sum of the harmfulness weights of the k pieces of intelligence and divide by the value of k, and the harmfulness weight depends on the harmfulness weights corresponding to different threat levels of the predefined intelligence to obtain the harmfulness index; The scheme layer judgment matrix calculation module is used to traverse threat intelligence sources for each evaluation index. When a threat intelligence source is traversed, it is used as a benchmark. According to the importance of the benchmark threat intelligence source compared with each threat intelligence source and the preset mapping relationship between different importance levels and importance scale values, the importance scale value of the benchmark threat intelligence source compared with each threat intelligence source is obtained. After the traversal ends, an n-order scheme layer judgment matrix is obtained; The scheme layer weight matrix calculation module is used to extract the first eigenvector and normalize the n×n scheme layer judgment matrices corresponding to m evaluation indicators respectively, and obtain the 1×n first-level single sorting weight matrices corresponding to m evaluation indicators respectively; The hierarchical total sorting weight calculation module is used to calculate the hierarchical total sorting weight of each threat intelligence source for intelligence source evaluation according to the second-level single sorting weight matrix of the preset m evaluation indicators for intelligence source evaluation and the first-level single sorting weight matrix of the 1 n-order first-level single sorting weight matrix; An intelligence source evaluation module, configured to perform an intelligence source evaluation based on the hierarchical total sorting weight of each threat intelligence source for the intelligence source evaluation.

8. An information source evaluation device, characterized in that, It includes at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 - 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions run on a computer, the computer is made to execute the method according to any one of claims 1 - 6.

Images