A method, device and apparatus for evaluating data source quality

Abstract

The present application discloses a method, device and equipment for evaluating the quality of a data source. By comprehensively considering the statistical information and confidence coefficients of multiple data sources, and calculating the evaluation scores of multiple dimensions of each data source, an evaluation result that can comprehensively reflect the data quality of each data source is obtained, and a conflict resolution processing algorithm is provided to provide highly reliable data results. Compared with the current single evaluation index for data sources, and the evaluation of the quality of a certain data source does not consider other data sources and only uses the data of the data source as the evaluation basis, resulting in a relatively one-sided evaluation result and a low reference value, and even misleading the staff, the accuracy of the evaluation results of the data source is effectively improved, and a reliable basis is provided for the staff to analyze and study the data in the data source, thereby improving the efficiency and accuracy of security threat detection.

Description

Technical Field

[0001] The present application relates to the field of security threat technology, and in particular to a method, device and equipment for evaluating the quality of a data source. Background Art

[0002] Each entity (such as a company) generates a lot of data, and analyzing the data of each entity is one of the important tasks in the field of network security. In the field of network security, an entity that generates data can be called a data source (also called an intelligence source), and a piece of data generated by each data source can be called a piece of intelligence. Usually, an entity adds data from multiple data sources to the security protection network of the entity through self-production or procurement, and supports network security protection work by analyzing and studying the data in the security protection network.

[0003] Since the data quality of different data sources varies, it is necessary to evaluate the data sources in order to provide a reference for the staff who analyze and study the data. At present, the evaluation indicators for data sources are relatively single, which makes the evaluation results relatively one-sided. Not only can they not provide a high reference value for the staff's analysis and research work, but inaccurate evaluation results can easily mislead the staff's research and judgment.

[0004] Based on this, there is an urgent need to provide a technical solution that can accurately evaluate the quality of the data source, improve the accuracy of the evaluation results of the data source, and provide a reliable basis for the staff to analyze and study the data in the security protection network. Summary of the invention

[0005] The embodiments of the present application provide a method, device and equipment for evaluating the quality of a data source, which can accurately evaluate the quality of a data source and provide a reliable basis for staff to analyze and study the data in the data source, thereby improving the efficiency and accuracy of security threat detection.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating the quality of a data source, including:

[0007] Obtain statistical information and confidence coefficients for each data source;

[0008] Calculating a score set for each of the data sources according to the statistical information and confidence coefficient of each of the data sources, the score set including at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set for each of the data sources are the same;

[0009] For each of the data sources, an evaluation result of the data source is calculated based on the score set of the data source, and the evaluation result of each data source is used to characterize the data quality of the data source.

[0010] Optionally, the data sources include a first data source and a second data source, and obtaining statistical information and a confidence coefficient of each data source includes:

[0011] Acquire first statistical information and a first confidence coefficient of the first data source, and acquire second statistical information and a second confidence coefficient of the second data source;

[0012] The calculating, according to the statistical information and confidence coefficient of each data source, a score set of each data source in the data sources includes:

[0013] According to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, a first score set of the first data source is calculated, and a second score set of the second data source is calculated, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set;

[0014] The step of calculating, for each of the data sources, an evaluation result of the data source based on the score set of the data source, includes:

[0015] A first evaluation result of the first data source is calculated based on the first score set, and a second evaluation result of the second data source is calculated based on the second score set, wherein the first evaluation result is used to characterize data quality of the first data source, and the second evaluation result is used to evaluate data quality of the second data source.

[0016] Optionally, the method further comprises:

[0017] Determining, according to the first evaluation result and the second evaluation result, that the data quality of the first data source is high;

[0018] Based on the data of the first data source, the data in the first data source and the data in the second data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis.

[0019] Optionally, the method further comprises:

[0020] The threat level of the threat indicator IOC is determined based on the first data source.

[0021] Optionally, the aggregating the data in the first data source and the data in the second data source based on the data in the first data source to obtain the integrated data includes:

[0022] Based on the data of the first data source, data in the first data source and the second data source are integrated, supplemented and conflict resolved to obtain the comprehensive data.

[0023] Optionally, the first data source and the second data source are data sources that have passed whitelist verification.

[0024] Optionally, if there is data update in the first data source, the method further includes:

[0025] Obtaining updated third statistical information and a third confidence coefficient of the first data source;

[0026] According to the second statistical information, the second confidence coefficient, the third statistical information and the third confidence coefficient, a third score set of the first data source is calculated, and a fourth score set of the second data source is calculated, the third score set includes at least two of the following scores: a third contribution score, a third matching score, a third applicability score, a third richness score, a third difference score and a third timeliness score, and the fourth score set includes at least two of the following scores: a fourth contribution score, a fourth matching score, a fourth applicability score, a fourth richness score, a fourth difference score and a fourth timeliness score; the number of scores included in the third score set and the performance indicators involved are the same as those of the fourth score set;

[0027] A third evaluation result of the first data source is calculated based on the third score set, and a fourth evaluation result of the second data source is calculated based on the fourth score set, the third evaluation result is used to characterize the data quality of the updated first data source, and the fourth evaluation result is used to evaluate the data quality of the second data source.

[0028] Optionally,

[0029] The calculating a first evaluation result of the first data source based on the first score set includes:

[0030] Calculate the average of the scores in the first score set, and record it as the first evaluation result;

[0031] The calculating a second evaluation result of the second data source based on the second score set includes:

[0032] An average value of the scores in the second score set is calculated and recorded as the second evaluation result.

[0033] Optionally, the data sources include a first data source, a second data source, and a third data source, and obtaining statistical information and a confidence coefficient of each data source includes:

[0034] Obtaining first statistical information and a first confidence coefficient of the first data source, obtaining second statistical information and a second confidence coefficient of the second data source, and obtaining fourth statistical information and a fourth confidence coefficient of the third data source;

[0035] The calculating, according to the statistical information and confidence coefficient of each data source, a score set of each data source in the data sources includes:

[0036] According to the first statistical information, the first confidence coefficient, the second statistical information, the second confidence coefficient, the fourth statistical information and the fourth confidence coefficient, a first score set for the first data source is calculated, a second score set for the second data source is calculated, and a fifth score set for the third data source is calculated, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score, and the fifth score set includes at least two of the following scores: a fifth contribution score, a fifth matching score, a fifth applicability score, a fifth richness score, a fifth difference score and a fifth timeliness score, and the first score set, the second score set and the fifth score set have the same number of scores and the same performance indicators involved;

[0037] The step of calculating, for each of the data sources, an evaluation result of the data source based on the score set of the data source, includes:

[0038] A first evaluation result of the first data source is calculated based on the first score set, a second evaluation result of the second data source is calculated based on the second score set, and a fifth evaluation result of the third data source is calculated based on the fifth score set, wherein the first evaluation result is used to characterize data quality of the first data source, the second evaluation result is used to evaluate data quality of the second data source, and the fifth evaluation result is used to characterize data quality of the third data source.

[0039] Optionally, the method further comprises:

[0040] According to the first evaluation result, the second evaluation result, and the fifth evaluation result, determining that the data quality of the first data source is higher than the data quality of the second data source, and the data quality of the second data source is higher than the data quality of the third data source;

[0041] Based on the data of the first data source, the data in the first data source, the data in the second data source and the data in the third data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis. During the aggregation process, the aggregation operation is performed on the data in multiple data sources in order from high to low data quality indicated by the evaluation results.

[0042] In a second aspect, an embodiment of the present application further provides a data source quality assessment device, including:

[0043] A first acquisition unit, used to acquire statistical information and confidence coefficients of each data source;

[0044] a first calculation unit, configured to calculate a score set of each of the data sources according to the statistical information and confidence coefficient of each of the data sources, the score set including at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set of each of the data sources are the same;

[0045] The second calculation unit is used to calculate the evaluation result of each data source among the data sources based on the score set of the data source, and the evaluation result of each data source is used to characterize the data quality of the data source.

[0046] Optionally, the data sources include a first data source and a second data source.

[0047] The first acquisition unit is specifically used to: acquire first statistical information and a first confidence coefficient of the first data source, and acquire second statistical information and a second confidence coefficient of the second data source;

[0048] The first calculation unit is specifically configured to calculate a first score set of the first data source and a second score set of the second data source according to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set;

[0049] The second calculation unit is specifically used to: calculate a first evaluation result of the first data source based on the first score set, and calculate a second evaluation result of the second data source based on the second score set, wherein the first evaluation result is used to characterize the data quality of the first data source, and the second evaluation result is used to evaluate the data quality of the second data source

[0050] Optionally, the device further comprises:

[0051] a first determining unit, configured to determine, according to the first evaluation result and the second evaluation result, that the data quality of the first data source is high;

[0052] The first aggregation unit is used to aggregate the data in the first data source and the data in the second data source based on the data in the first data source to obtain comprehensive data, and the comprehensive data is used for network security analysis.

[0053] Optionally, the device further comprises:

[0054] The second determining unit is configured to determine a threat level of a threat indicator IOC based on the first data source.

[0055] Optionally, the polymerization unit is specifically used for:

[0056] Based on the data of the first data source, data in the first data source and the second data source are integrated, supplemented and conflict resolved to obtain the comprehensive data.

[0057] Optionally, the first data source and the second data source are data sources that have passed whitelist verification.

[0058] Optionally, if there is data update in the first data source, the device further includes:

[0059] A second acquisition unit, configured to acquire updated third statistical information and a third confidence coefficient of the first data source;

[0060] a third calculation unit, configured to calculate a third score set of the first data source and a fourth score set of the second data source according to the second statistical information, the second confidence coefficient, the third statistical information and the third confidence coefficient, wherein the third score set includes at least two of the following scores: a third contribution score, a third matching score, a third applicability score, a third richness score, a third difference score and a third timeliness score, and the fourth score set includes at least two of the following scores: a fourth contribution score, a fourth matching score, a fourth applicability score, a fourth richness score, a fourth difference score and a fourth timeliness score; the number of scores included in the third score set and the performance indicators involved are the same as those of the fourth score set;

[0061] a fourth calculation unit, configured to calculate a third evaluation result of the first data source based on the third score set, and calculate a fourth evaluation result of the second data source based on the fourth score set, wherein the third evaluation result is used to characterize the data quality of the updated first data source, and the fourth evaluation result is used to evaluate the data quality of the second data source.

[0062] Optionally,

[0063] The second computing unit comprises:

[0064] a first calculation subunit, configured to calculate an average of the scores in the first score set, and record the average as the first evaluation result;

[0065] The second calculation subunit is used to calculate the average value of each score in the second score set, which is recorded as the second evaluation result.

[0066] Optionally, the data sources include a first data source, a second data source and a third data source.

[0067] The first acquisition unit is specifically used to: acquire first statistical information and a first confidence coefficient of the first data source, acquire second statistical information and a second confidence coefficient of the second data source, and acquire fourth statistical information and a fourth confidence coefficient of the third data source;

[0068] The first calculation unit is specifically configured to calculate a first score set for the first data source, a second score set for the second data source, and a fifth score set for the third data source according to the first statistical information, the first confidence coefficient, the second statistical information, the second confidence coefficient, the fourth statistical information, and the fourth confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score, and a first timeliness score, the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score, and a second timeliness score, and the fifth score set includes at least two of the following scores: a fifth contribution score, a fifth matching score, a fifth applicability score, a fifth richness score, a fifth difference score, and a fifth timeliness score, and the first score set, the second score set, and the fifth score set have the same number of scores and the same performance indicators involved;

[0069] The second calculation unit is specifically configured to calculate a first evaluation result of the first data source based on the first score set, calculate a second evaluation result of the second data source based on the second score set, and calculate a fifth evaluation result of the third data source based on the fifth score set, wherein the first evaluation result is used to characterize data quality of the first data source, the second evaluation result is used to evaluate data quality of the second data source, and the fifth evaluation result is used to characterize data quality of the third data source.

[0070] Optionally, the device further comprises:

[0071] a third determining unit, configured to determine, according to the first evaluation result, the second evaluation result, and the fifth evaluation result, that the data quality of the first data source is higher than the data quality of the second data source, and that the data quality of the second data source is higher than the data quality of the third data source;

[0072] The second aggregation unit is used to aggregate the data in the first data source, the data in the second data source and the data in the third data source based on the data of the first data source to obtain comprehensive data, wherein the comprehensive data is used for network security analysis. During the aggregation process, the aggregation operation is performed on the data in the multiple data sources in the order of data quality indicated by the evaluation results from high to low.

[0073] It should be noted that the specific implementation method and effect achieved by the data source quality evaluation device provided in the second aspect can refer to the description of the relevant embodiment of the data source quality evaluation method shown in the first aspect.

[0074] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising a processor and a memory:

[0075] The memory is used to store computer programs;

[0076] The processor is used to execute the method provided in the first aspect according to the computer program.

[0077] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method provided in the first aspect above.

[0078] It can be seen that the embodiments of the present application have the following beneficial effects:

[0079] An embodiment of the present application provides a method for evaluating the quality of a data source, which method may, for example, include: a data source quality evaluation device obtains statistical information and a confidence coefficient of each data source; based on the statistical information and the confidence coefficient of each data source, a score set of each of the data sources is calculated, the score set includes at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set of each of the data sources are the same; for each of the data sources, an evaluation result of the data source is calculated based on the score set of the data source, and the evaluation result of each data source is used to characterize the data quality of the data source. It can be seen that the method provided in the embodiment of the present application comprehensively considers the statistical information and confidence coefficients of multiple data sources, calculates the evaluation scores of multiple dimensions of each data source, and obtains an evaluation result that can comprehensively reflect the data quality of each data source. Compared with the current evaluation of data sources, which has a single indicator and evaluates the quality of a certain data source without considering other data sources and only uses the data of the data source as the basis for evaluation, resulting in a one-sided evaluation result with low reference value and even misleading the staff, it effectively improves the accuracy of the evaluation results of the data source, provides a reliable basis for the staff to analyze and study the data in the data source, thereby improving the efficiency and accuracy of security threat detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0081] Figure 1A schematic diagram of a flow chart of a method for evaluating data source quality provided in an embodiment of the present application;

[0082] Figure 2 A schematic diagram of conflict resolution provided in an embodiment of the present application;

[0083] Figure 3 A schematic diagram showing evaluation results of a data source provided in an embodiment of the present application;

[0084] Figure 4 A schematic diagram of the structure of a data source quality assessment device provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, it should be noted that, for ease of description, only the parts related to the present application are shown in the accompanying drawings, not all structures.

[0087] With the continuous improvement of the accuracy and quality requirements for security threat detection capabilities, the production, operation and sharing of intelligence libraries have also become a manifestation of the basic capabilities of various companies. Companies are more likely to add open source threat intelligence to the company's security protection network through self-production and procurement, relying on multi-source intelligence to provide a richer data foundation for security threat detection. However, multi-source intelligence comes from different companies, and the quality of intelligence from different companies varies, which brings certain difficulties to the analysis work of staff. It is necessary to evaluate the intelligence quality of multi-source intelligence to improve the accuracy of security threat detection. It should be noted that the intelligence mentioned in the embodiments of the present application refers to data that can be detected by security threats. The intelligence source is the subject (such as a company) that provides intelligence (or data), and the intelligence library is the carrier for storing intelligence.

[0088] At present, the indicators for evaluating intelligence are relatively simple, which makes the evaluation results rather one-sided. Not only can they not provide a high reference value for the staff's analysis and research work, but inaccurate evaluation results can easily mislead the staff's judgment.

[0089] Based on this, an embodiment of the present application provides a method that can accurately evaluate the quality of a data source. The data source quality evaluation device can obtain statistical information and confidence coefficients of multiple data sources, and calculate a score set for each data source based on the statistical information and confidence coefficients of the multiple data sources. The score set includes at least two of the following scores: contribution score, matching score, applicability score, richness score, difference score and timeliness score; thus, the data source quality evaluation device can calculate the evaluation results of each data source based on the score set of each data source, and the evaluation result of each data source is used to characterize the data quality of the data source. Taking multiple data sources including a first data source and a second data source as an example, the method provided in an embodiment of the present application may include, for example: a method for evaluating data source quality, including: obtaining first statistical information and a first confidence coefficient of the first data source, and obtaining second statistical information and a second confidence coefficient of the second data source; calculating a first score set of the first data source and a second score set of the second data source according to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, the first score set including at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set including at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set; calculating a first evaluation result of the first data source based on the first score set, and calculating a second evaluation result of the second data source based on the second score set, the first evaluation result being used to characterize the data quality of the first data source, and the second evaluation result being used to evaluate the data quality of the second data source.

[0090] It can be seen that the method provided in the embodiment of the present application calculates the scores of each data source in multiple dimensions (at least two dimensions of contribution, matching, applicability, richness, difference and timeliness) by combining the statistical information and confidence coefficients of other data sources, and obtains the evaluation result of the intelligence source by integrating the scores of multiple dimensions. Since the evaluation result takes into account the conditions of other data sources and the conditions of multiple different dimensions, the evaluation result can accurately reflect the data quality of the data source, save workload for subsequent security threat detection and analysis and research by staff, provide a reliable basis, and thus improve the accuracy of security threat detection.

[0091] It should be noted that the subject implementing the data source quality assessment method may be a data source quality assessment device provided in the embodiment of the present application, and the device may be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiment of the present application may be any device capable of implementing the data source quality assessment method in the embodiment of the present application.

[0092] To facilitate understanding of the specific implementation of the data source quality assessment method provided in the embodiment of the present application, it will be described below with reference to the accompanying drawings.

[0093] Figure 1 A schematic diagram of a method for evaluating data source quality provided in an embodiment of the present application. The method is applied to a device for evaluating data source quality. Figure 1 As shown, the method may include the following S101 to S103:

[0094] S101, obtaining statistical information and confidence coefficients of each data source.

[0095] Among them, statistical information may refer to the information in the data source required for the calculation in S102, for example, it may include but not be limited to: the total number of data sources, the number of various entity objects in the data source, the number of data sources updated on the day, the number of various entity objects in the data updated on the day, etc. Among them, entity objects may include but are not limited to: domain name (domainname), Internet Protocol (Internet Protocol, IP) address, uniform resource locator (uniform resource locator, URL), md5 information digest algorithm, secure hash algorithm (Secure Hash Algorithm, sha)-1. For the input of different entity objects, the evaluation system, confidence and multi-dimensional comprehensive weight calculation can be performed, and after entering the evaluation system for scoring, reliable data is output, and analysis and judgment output is provided based on the data quality of different sources.

[0096] Since confidence has the greatest impact on the evaluation of data sources, it is preferred to use the confidence coefficient that reflects confidence as the calculation input. In one case, the confidence coefficient can refer to the confidence value set for the data source, which can usually be set to 1 to 5. In another case, in order to make the scores obtained by subsequent use of the confidence coefficient tend to be standardized, the confidence coefficient can also be the value obtained by converting the set confidence according to a preset conversion relationship. Among them, the preset conversion relationship can be, for example, the following formula (1):

[0097] confidence_ratio=confidence*0.1+0.5...Formula (1)

[0098] Among them, confidence is the confidence level set for the data source, and confidence_ratio is the confidence coefficient.

[0099] It should be noted that the confidence levels of different data sources may be the same or different, that is, the confidence coefficients of different data sources may be the same or different.

[0100] S102, calculating a score set for each of the data sources based on the statistical information and confidence coefficient of the data sources, the score set including at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores included in the score set for each of the data sources and the performance indicators involved are the same.

[0101] In the embodiment of the present application, the data in the data source is deduplicated, normalized, formatted, and factored through a quality assessment algorithm, and at least two of the following six assessment dimensions are evaluated: timeliness, richness, diversity, contribution, matching, and applicability. The six dimensions may also be referred to as the six characteristics of threat intelligence, which are indicators of the six characteristics of threat intelligence in the intelligence quality evaluation system established in the embodiment of the present application, and different algorithm inputs are provided for different measurement indicators.

[0102] The contribution score is a measure of the richness of data source delivery by evaluating the proportion of data pushed by each data source to the total amount of data from all data sources. For example, the contribution score can be calculated using the following formula (2):

[0103]

[0104] In formula (2), first calculate the percentage of the number of data items m(i) of data source i to the total number of data items of all n data sources, then multiply the obtained percentage by 100 and add the basic score C, and finally multiply the obtained sum by the confidence coefficient to obtain the contribution score.

[0105] The matching score is to measure the matching of data sources by evaluating the proportion of the data types pushed by each data source to all the data types of the data source. For example, the data types pushed by a data source are domain names accounting for 20% and IP addresses accounting for 80%. For example, the matching score can be calculated by the following formula (3):

[0106]

[0107] In formula (3), it is assumed that there are N data sources in total. For data source i, each data source has m(i) types of intelligence, and its j-th type of data is m(i,j). Calculate the ratio of each type of data of data source i to the sum of the number of data of this type in all data sources, multiply the ratio by 100, add the results of these multiplications of 100 for all intelligence sources, add C, and then multiply by the confidence coefficient to obtain the matching score. It should be noted that the matching score may exceed 100. For example, there are 1000 URL data in total, and data source i accounts for 999 of them. Then the matching score of data source i is (99.9+C)*confidence_ratio. Depending on the value of C and the specific situation of the confidence coefficient, the matching score obtained is likely to exceed 100.

[0108] The applicability score is a measure of the applicability of a data source by evaluating the proportion of the data hit by the query in each data source. For example, the applicability score can be calculated using the following formula (4):

[0109]

[0110] In formula (4), N data are randomly selected, and data source i hits H(i) data of M categories. The number of hits in each category is H(i,j). The sum of the ratios of the number of hits in each category to the number of selected data of that category N(j) is calculated, multiplied by 100, added to C, and then multiplied by the confidence coefficient to obtain the applicability score.

[0111] The richness score refers to the quality assessment of different threat data sources based on the richness of auxiliary information in each data source, where the auxiliary information includes but is not limited to: Secure Socket Layer (SSL) certificates, passive domain name resolution data, digital signatures, etc. For example, the richness score can be calculated using the following formula (5):

[0112] O4=(O(n)+4(1+P(n)+C)*confidence_ratio(i)…Formula (5)

[0113] In formula (5), O(n) is the number of fields included in the data source, and P(n) is the median number of tags of all threat indicators (Indicators of Compromise, IOC) of the data source.

[0114] The difference score is a measure of the independence of the data source's output capacity and the technical characteristics and regionality of the data source's production organization through the overlap ratio of the data between the data sources. For example, the difference score can be calculated by the following formula (6):

[0115]

[0116] In formula (6), m(i) is the IOC set of a data source i. The ratio of the size of the IOC set to the size of the IOC set of all data sources is multiplied by 100, added to C, and then multiplied by the confidence coefficient to obtain the difference score.

[0117] The timeliness score is a measure of the data output efficiency of a data source by measuring the data update frequency and the proportion of first-released data results of each data. For example, the timeliness score can be calculated using the following formula (7):

[0118]

[0119] In formula (7), the number of data items m(i) updated on the same day by data source i accounts for the percentage of the total number of data items updated on the same day by all n data sources, multiplied by 100, added to C, and then multiplied by the confidence coefficient to obtain the timeliness score.

[0120] Through the above explanation of the meaning of the scores of each dimension and the exemplary description of the calculation method, the data source quality evaluation device can calculate the score set of each data source based on the statistical information and confidence coefficient of each data source obtained in S101, and is ready to determine the evaluation results of each data source.

[0121] S103, for each of the data sources, calculating an evaluation result of the data source based on the score set of the data source, wherein the evaluation result of each data source is used to characterize the data quality of the data source.

[0122] Taking the data sources including the first data source and the second data source as an example, the above S101 may include: obtaining first statistical information and a first confidence coefficient of the first data source, and obtaining second statistical information and a second confidence coefficient of the second data source; S102 may include: calculating a first score set of the first data source and a second score set of the second data source according to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set; S103 may include: calculating a first evaluation result of the first data source based on the first score set, and calculating a second evaluation result of the second data source based on the second score set, wherein the first evaluation result is used to characterize the data quality of the first data source, and the second evaluation result is used to evaluate the data quality of the second data source. Calculating the first evaluation result of the first data source based on the first score set in S103 may include: calculating the average of the scores in the first score set, and recording it as the first evaluation result. Calculating the second evaluation result of the second data source based on the second score set in S103 may include: calculating the average of the scores in the second score set, and recording it as the second evaluation result.

[0123] As an example, the method of calculating the evaluation result of the data source may include, for example, removing the highest score and the lowest score from the six scores and taking the average of the remaining scores. For example, the evaluation result may be expressed as: M = [(O1+O2+O3+O4+O5+O6)-min(O1,O2,O3,O4,O5,O6)-max(O1,O2,O3,O4,O5,O6)] / 6, where M is the evaluation result of the data source.

[0124] It should be noted that for newly connected data sources, a default value will be assigned to the six scores to avoid extreme values ​​that may cause calculation failure.

[0125] For example, suppose there are two data sources A and B, and the confidence of each data source is set to 4, then: confidence coefficient confidence_ratio = 4*0.1+0.5 = 0.9. At the same time, set the initial score C = 50. If it is assumed that the IP type of data source A and the IP type of data source B have 5,000 data in total. Data source A updates 4,000 data on the same day, of which 2,200 are IP type data and 1,800 are domain name type data. The total number of data sources A is 20,000, the total number of IP type data is 12,000, and the total number of domain name type data is 8,000. The data sample of data source A is as follows:

[0126] "2021-09-16","2022-03-08","imabug.com","APT","doc"

[0127] "2022-01-31","2022-03-08","bugcoop.com","APT","doc"

[0128] Data source B updates 8,000 pieces of data on the day, of which 5,000 are IP type data and 3,000 are domain name type data. The total number of data sources in data source B is 80,000, of which 50,000 are IP type data and 30,000 are domain name type data. The data sample of data source B is as follows:

[0129] #id,dateadded,url,url_status,threat,tags,urlhaus_link,reporter

[0130] "143","2022-03-07

[0131] 06:49:14","http: / / 125.53.125.15:32208 / Xozi.b","online","malware_download","elf,Mozi","https: / / urlhaus.absuse.ch / url / 14743 / ","lrzl_urlhaus"

[0132] "144","2022-03-07

[0133] 06:49:24","http: / / 125.53.125.15:32208 / Dozi.b","online","malware_download","elf,Mozi","https: / / urlhaus.absuse.ch / url / 14742 / ","lrzl_urlhaus"

[0134] 1000 data are randomly selected from each data source as sample IOC data to calculate the data hit rate (i.e., applicability score). Assume that 500 of the 1000 samples hit data source A and 800 hit data source B. Then, based on the calculation of the six scores using the above formulas (2) to (7), we can obtain:

[0135] In data source A,

[0136] AO1=((20000 / (20000+80000)*100)+50)*0.9=63

[0137] AO2=((12000 / (12000+50000)+8000 / (8000+30000))*100+50)*0.9=81.36

[0138] AO3=(500 / 20000*100+50)*0.9=47.25

[0139] AO4=(5+4*(1+1)+50)*0.9=52.2

[0140] AO5=(5000 / 20000*100+50)*0.9=56.7

[0141] AO6=(4000 / (4000+8000)*100+50)*0.9=75

[0142] BData source.

[0143] BO1=((80000 / (20000+80000)*100)+50)*0.9=117

[0144] BO2=((50000 / (12000+50000)+30000 / (8000+30000))*100+50)*0.9=188

[0145] BO3=(800 / 80000*100+50)*0.9=45.9

[0146] BO4 = (8 + 4 * (1 + 2) + 50) * 0.9 = 63

[0147] BO5=(5000 / 100000*100+50)*0.9=49.5

[0148] BO6=(8000 / (4000+8000)*100+50)*0.9=105

[0149] Then, the evaluation result M1 of data source A can be equal to [(AO1+AO2+AO3+AO4+AO5+AO6)-min(AO1,AO2,AO3,AO4,AO5,AO6)-max(AO1,AO2,AO3,AO4,AO5,AO6)] / 6=[63+81.36+47.25+52.2+56.7+75-(47.25-81.36)] / 6=41.45. The evaluation result M2 of the B data source can be equal to [(BO1+BO2+BO3+BO4+BO5+BO6)-min(BO1,BO2,BO3,BO4,BO5,BO6)-max(BO1,BO2,BO3,BO4,BO5,BO6)] / 6=117+188+45.9+63+49.5+105-(45.9-188) / 6=55.75. It can be seen that through the method provided in the embodiment of the present application, the quality scores of the A data source and the B data source are calculated as follows: the A data source is: 44.15 points, and the B data source is: 55.75 points.

[0150] It should be noted that when the evaluation result of the data source is a score, the higher the score, the higher the data quality of the data source.

[0151] It should be noted that timeliness is very important for the data quality assessment of the data source. Therefore, in the embodiment of the present application, after each data source obtains updated data, a quality assessment will be performed in real time. If a data source is updated more frequently, it means that the timeliness of the score is higher. During the assessment period, the algorithm provided in the embodiment of the present application is used to implement a comprehensive aggregation evaluation so that the score evaluated by each data source is timely and effective.

[0152] As an example, after S103, if there is data update in the first data source, the method may further include: obtaining third statistical information and a third confidence coefficient of the updated first data source; calculating a third score set of the first data source and a fourth score set of the second data source according to the second statistical information, the second confidence coefficient, the third statistical information and the third confidence coefficient, the third score set including at least two of the following scores: a third contribution score, a third matching score, a third applicability score, a third richness score, a third difference score and a third timeliness score, and the fourth score set including at least two of the following scores: a fourth contribution score, a fourth matching score, a fourth applicability score, a fourth richness score, a fourth difference score and a fourth timeliness score; the number of scores included in the third score set and the performance indicators involved are the same as those of the fourth score set; calculating a third evaluation result of the first data source based on the third score set, and calculating a fourth evaluation result of the second data source based on the fourth score set, the third evaluation result being used to characterize the data quality of the updated first data source, and the fourth evaluation result being used to evaluate the data quality of the second data source. In this way, the most valuable evaluation results can be obtained, which provides a reliable basis for subsequent processing of data from the data source based on the evaluation results.

[0153] Taking the data sources including the first data source, the second data source and the third data source as an example, the above S101 may include: obtaining first statistical information and a first confidence coefficient of the first data source, obtaining second statistical information and a second confidence coefficient of the second data source, and obtaining fourth statistical information and a fourth confidence coefficient of the third data source; the above S102 may include: calculating a first score set of the first data source, calculating a second score set of the second data source, and calculating a fifth score set of the third data source according to the first statistical information, the first confidence coefficient, the second statistical information, the second confidence coefficient, the fourth statistical information and the fourth confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first suitability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score The fifth score set includes at least two of the following scores: a fifth contribution score, a fifth match score, a fifth suitability score, a fifth richness score, a fifth difference score and a fifth timeliness score, and the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set and the fifth score set; the above S103 may include: calculating a first evaluation result of the first data source based on the first score set, calculating a second evaluation result of the second data source based on the second score set, and calculating a fifth evaluation result of the third data source based on the fifth score set, the first evaluation result is used to characterize the data quality of the first data source, the second evaluation result is used to evaluate the data quality of the second data source, and the fifth evaluation result is used to characterize the data quality of the third data source.

[0154] The method may also include: determining, based on the first evaluation result, the second evaluation result and the fifth evaluation result, that the data quality of the first data source is higher than the data quality of the second data source, and the data quality of the second data source is higher than the data quality of the third data source; based on the data of the first data source, aggregating the data in the first data source, the data in the second data source and the data in the third data source to obtain comprehensive data, wherein the comprehensive data is used for network security analysis, and during the aggregation process, the aggregation operation is performed on the data in multiple data sources in the order of data quality indicated by the evaluation results from high to low.

[0155] It can be seen that through the method provided in the embodiment of the present application, the parameters of multiple data sources can be combined to calculate for each data source an evaluation result that ultimately reflects the data quality in each data source, and aggregation can be performed based on the data quality indicated by the evaluation result, making it possible to provide a complete and reference-worthy comprehensive data, providing a data basis for network security analysis and judgment based on the comprehensive data.

[0156] In some implementations, after S103, the present application embodiment also proposes to fuse data from multiple data sources based on five methods: blacklist, whitelist, confidence, timeliness, threat type, and threat level, so as to reduce the workload and improve the accuracy for subsequent analysis and judgment of the data. The following description is based on an example in which each data source includes a first data source and a second data source.

[0157] As an example, the method provided in the embodiment of the present application may also include: if it is determined that the data quality of the first data source is high based on the first evaluation result and the second evaluation result; then, based on the data of the first data source, the data in the first data source and the data in the second data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis. Alternatively, if it is determined that the data quality of the second data source is high based on the first evaluation result and the second evaluation result; then, based on the data of the second data source, the data in the second data source and the data in the first data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis. For the above-mentioned examples of data source A and data source B, the B data source corresponding to M2 with a higher evaluation score will be used as the basic data for merging and digestion. The data quality of data source B is higher, and the output field content and aggregation results will be more biased towards data source B.

[0158] If it is determined based on the first evaluation result and the second evaluation result that the data quality of the first data source is high, then based on the data of the first data source, the data in the first data source and the data in the second data source are aggregated to obtain comprehensive data, which may include, for example: based on the data of the first data source, data integration, supplementation and conflict resolution are performed on the first data source and the second data source to obtain the comprehensive data.

[0159] In the embodiment of the present application, the process of conflict resolution based on the five threat intelligence fusion methods of blacklist, whitelist, timeliness, confidence, richness, difference, hit rate, etc. can be as follows: Figure 2 As shown, it includes the following S1 to S5:

[0160] S1, black and white list matching, that is, all entity IOCs will be filtered through the system preset white list and user-defined white list first, and then the black and white pre-processing analysis will be performed. Then, the first data source and the second data source in S101 can be understood as data sources that have passed the white list verification.

[0161] S2, timeliness guarantee, that is, the data of the data source may be updated regularly, and all data sources will be re-scored based on the latest data to ensure timeliness.

[0162] S3, confidence intervention, that is, users can preset confidence scores for each data source and conduct manual intervention. Confidence will affect the six-characteristic scores of the data, with custom intervention scores as priority, which will ultimately affect the optimal data source selected during conflict resolution and the weight of each data source during data enrichment. It should be noted that when data sources are enriched, the data source with the highest evaluation result obtained based on the six-characteristic score is usually used as the optimal solution, and the remaining data sources are sorted from high to low according to the evaluation results to supplement the fields.

[0163] S4, threat level determination, judging the threat level of the IOC based on the data source with the highest evaluation result score. For example, assuming that the data quality of the first data source is high based on the first evaluation result and the second evaluation result, then the threat level of the threat indicator IOC is determined based on the first data source.

[0164] S5, threat type determination, fusion of threat type information of data sources during conflict resolution (including integration, filling, and conflict resolution).

[0165] For example, suppose IOC 78.130.133.60 has three data sources: AB C. The fields of the three data sources are different, including inconsistent field descriptions, different field richness, etc. The evaluation results of data sources A, B, and C are ranked from high to low: the evaluation result of data source A: 80 points, the evaluation result of data source B: 60 points, and the evaluation result of data source C: 40 points.

[0166] Example 1: For "Threat Level", the field name of data source A is Risk, the field name of data source B is severity, and there is no corresponding field in data source C. Based on the name of the field in data source A, the fields corresponding to data sources B and C are uniformly named Risk, and the value of the Risk field in data source A is assigned to the fields corresponding to data sources B and C.

[0167] Example 2: For "threat level", data source A does not have this related field, data source B corresponds to the field severity, and data source C corresponds to the field risk. Then, first, the corresponding field can be mapped to risk, and the data of data source A with the highest overall rating is used as the basic data. If data source A does not contain the risk field, then from data source B with the highest evaluation result containing this field, the value of the risk field in the final result is supplemented with the value of the risk field (original severity field) of data source B.

[0168] Example 3: For labels, data source A includes [a,b], data source B includes [b,c], and data source C has no labels. In this case, the corresponding fields can be mapped to tags, and then the union of the labels appearing in all data sources can be taken, that is, tags: [a,b,c].

[0169] For each IOC, a normalized and conflict-resolved data is obtained. The fields and values ​​included in the data are as follows:

[0170] {"source":"XXX",

[0171] "name":"RiskWarePK[Downloader] / Win32.DownloaderGuidePK.pwj",

[0172] "os":"Win32",

[0173] "md5":"65bd1ba313a36f779434a1acc1def8b2",

[0174] "malicious_type":["Network Worm"],

[0175] "sub_type":"Computer malicious program",

[0176] "description":"Sample",

[0177] "confidence":"low",

[0178] "tlp":"red",

[0179] "risk":"low",

[0180] "ioc":[{"type":"MD5","MD5":"65bd1ba313a36f779434a1acc1def8b2"}],

[0181] "block_impact":"1","country":"China","province":"Shanghai","city":"Shanghai",

[0182] "level":"3",

[0183] "submitter":"",

[0184] "tag":['a','b','c'],

[0185] "release_time":"2021-11-08",

[0186] "create_time"":"2021-10-2801:17:52",

[0187] "first_seen":"2021-10-2801:17:52",

[0188] "last_seen":"2021-10-2811:17:52",}

[0189] Finally, the data after conflict resolution from multiple data sources can be stored in the database, and a corresponding query interface can be provided to complete the conflict resolution of IOC data and the output of high-quality data.

[0190] It can be seen that through the method provided in the embodiment of the present application, not only can the evaluation results that can ultimately reflect the data quality in the data source be obtained, but also the scores of each dimension in multiple dimensions can be obtained. When a large number of data sources are connected, it has a very obvious effect when used. The conflict resolution information after business logic integration can reasonably display the evaluation results of different integrated data based on the six-property weight multidimensional algorithm.

[0191] like Figure 3 As shown, it is the evaluation result of the visualized data source provided in the embodiment of the present application. It can be seen that the method provided in the embodiment of the present application can clearly display the evaluation score of each connected data source through a visual interface, so that the staff performing threat analysis and security analysis can clearly see the comprehensive quality of the IOC and the situation in each dimension, making it possible to obtain high-value data, greatly improving the reliability and accuracy of the judgment, reducing the manpower cost of the staff, and increasing the work efficiency of the staff in threat analysis and threat judgment.

[0192] Accordingly, the embodiment of the present application also provides a data source quality evaluation device 400, such as Figure 4 As shown, the apparatus 400 may include:

[0193] A first acquisition unit 401 is used to acquire statistical information and confidence coefficients of each data source;

[0194] A first calculation unit 402 is used to calculate a score set of each of the data sources according to the statistical information and confidence coefficient of each of the data sources, where the score set includes at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set of each of the data sources are the same;

[0195] The second calculation unit 403 is used to calculate the evaluation result of each data source among the data sources based on the score set of the data source, and the evaluation result of each data source is used to characterize the data quality of the data source.

[0196] Optionally, the data sources include a first data source and a second data source.

[0197] The first acquisition unit 401 is specifically used to: acquire first statistical information and a first confidence coefficient of the first data source, and acquire second statistical information and a second confidence coefficient of the second data source;

[0198] The first calculation unit 402 is specifically configured to: calculate a first score set of the first data source and a second score set of the second data source according to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set;

[0199] The second calculation unit 403 is specifically used to: calculate a first evaluation result of the first data source based on the first score set, and calculate a second evaluation result of the second data source based on the second score set, wherein the first evaluation result is used to characterize the data quality of the first data source, and the second evaluation result is used to evaluate the data quality of the second data source

[0200] Optionally, the device 400 further includes:

[0201] a first determining unit, configured to determine, according to the first evaluation result and the second evaluation result, that the data quality of the first data source is high;

[0202] The first aggregation unit is used to aggregate the data in the first data source and the data in the second data source based on the data in the first data source to obtain comprehensive data, and the comprehensive data is used for network security analysis.

[0203] Optionally, the device 400 further includes:

[0204] The second determining unit is configured to determine a threat level of a threat indicator IOC based on the first data source.

[0205] Optionally, the polymerization unit is specifically used for:

[0206] Based on the data of the first data source, data in the first data source and the second data source are integrated, supplemented and conflict resolved to obtain the comprehensive data.

[0207] Optionally, the first data source and the second data source are data sources that have passed whitelist verification.

[0208] Optionally, if there is data update in the first data source, the apparatus 400 further includes:

[0209] A second acquisition unit, configured to acquire updated third statistical information and a third confidence coefficient of the first data source;

[0210] a third calculation unit, configured to calculate a third score set of the first data source and a fourth score set of the second data source according to the second statistical information, the second confidence coefficient, the third statistical information and the third confidence coefficient, wherein the third score set includes at least two of the following scores: a third contribution score, a third matching score, a third applicability score, a third richness score, a third difference score and a third timeliness score, and the fourth score set includes at least two of the following scores: a fourth contribution score, a fourth matching score, a fourth applicability score, a fourth richness score, a fourth difference score and a fourth timeliness score; the number of scores included in the third score set and the performance indicators involved are the same as those of the fourth score set;

[0211] a fourth calculation unit, configured to calculate a third evaluation result of the first data source based on the third score set, and calculate a fourth evaluation result of the second data source based on the fourth score set, wherein the third evaluation result is used to characterize the data quality of the updated first data source, and the fourth evaluation result is used to evaluate the data quality of the second data source.

[0212] Optionally,

[0213] The second computing unit comprises:

[0214] a first calculation subunit, configured to calculate an average of the scores in the first score set, and record the average as the first evaluation result;

[0215] The second calculation subunit is used to calculate the average value of each score in the second score set, which is recorded as the second evaluation result.

[0216] Optionally, the data sources include a first data source, a second data source and a third data source.

[0217] The first acquisition unit 401 is specifically used to: acquire first statistical information and a first confidence coefficient of the first data source, acquire second statistical information and a second confidence coefficient of the second data source, and acquire fourth statistical information and a fourth confidence coefficient of the third data source;

[0218] The first calculation unit 402 is specifically configured to calculate a first score set for the first data source, a second score set for the second data source, and a fifth score set for the third data source according to the first statistical information, the first confidence coefficient, the second statistical information, the second confidence coefficient, the fourth statistical information, and the fourth confidence coefficient, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score, and a first timeliness score, the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score, and a second timeliness score, and the fifth score set includes at least two of the following scores: a fifth contribution score, a fifth matching score, a fifth applicability score, a fifth richness score, a fifth difference score, and a fifth timeliness score, and the first score set, the second score set, and the fifth score set have the same number of scores and the same performance indicators involved;

[0219] The second calculation unit 403 is specifically configured to calculate a first evaluation result of the first data source based on the first score set, calculate a second evaluation result of the second data source based on the second score set, and calculate a fifth evaluation result of the third data source based on the fifth score set, wherein the first evaluation result is used to characterize data quality of the first data source, the second evaluation result is used to evaluate data quality of the second data source, and the fifth evaluation result is used to characterize data quality of the third data source.

[0220] Optionally, the device 400 further includes:

[0221] a third determining unit, configured to determine, according to the first evaluation result, the second evaluation result, and the fifth evaluation result, that the data quality of the first data source is higher than the data quality of the second data source, and that the data quality of the second data source is higher than the data quality of the third data source;

[0222] The second aggregation unit is used to aggregate the data in the first data source, the data in the second data source and the data in the third data source based on the data of the first data source to obtain comprehensive data, wherein the comprehensive data is used for network security analysis. During the aggregation process, the aggregation operation is performed on the data in the multiple data sources in the order of data quality indicated by the evaluation results from high to low.

[0223] In addition, the present application embodiment also provides an electronic device 500, such as Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502:

[0224] The memory 502 is used to store computer programs;

[0225] The processor 501 is used to execute the method provided in the embodiment of the present application according to the computer program.

[0226] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method provided by the embodiment of the present application.

[0227] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0228] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for system embodiments and device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, in which the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0229] The above is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the present application, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. A method for evaluating the quality of a data source, characterized in that: include: Obtain statistical information and confidence coefficients for each data source; Calculating a score set for each of the data sources according to the statistical information and confidence coefficient of each of the data sources, the score set including at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set for each of the data sources are the same; For each of the data sources, an evaluation result of the data source is calculated based on the score set of the data source, and the evaluation result of each data source is used to characterize the data quality of the data source; Among them, the contribution score refers to the proportion of data pushed by the data source to the total amount of data pushed by all data sources; the matching score refers to the proportion of data of any data type pushed by the data source to the total amount of data of this data type pushed by all data sources; the applicability score refers to the proportion of data corresponding to any data type that has passed the evaluation query and hit in the data source to the total amount of data of this data type pushed by the data source; the richness score refers to the richness of auxiliary information in each data source, and the auxiliary information includes one or more of secure socket protocol certificates, passive domain name resolution books, and digital signatures; the difference score refers to the overlap ratio of data between data sources; the timeliness score refers to the data update frequency and the proportion of first-released data results of each data source.

2. The method according to claim 1, characterized in that The data sources include a first data source and a second data source, and obtaining statistical information and a confidence coefficient of each data source includes: Acquire first statistical information and a first confidence coefficient of the first data source, and acquire second statistical information and a second confidence coefficient of the second data source; The calculating, according to the statistical information and confidence coefficient of each data source, a score set of each data source in the data sources includes: According to the first statistical information, the first confidence coefficient, the second statistical information and the second confidence coefficient, a first score set of the first data source is calculated, and a second score set of the second data source is calculated, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, and the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score; the number of scores included in the first score set and the performance indicators involved are the same as those of the second score set; The step of calculating, for each of the data sources, an evaluation result of the data source based on the score set of the data source, includes: A first evaluation result of the first data source is calculated based on the first score set, and a second evaluation result of the second data source is calculated based on the second score set, wherein the first evaluation result is used to characterize data quality of the first data source, and the second evaluation result is used to evaluate data quality of the second data source.

3. The method according to claim 2, characterized in that The method further comprises: Determining, according to the first evaluation result and the second evaluation result, that the data quality of the first data source is high; Based on the data of the first data source, the data in the first data source and the data in the second data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis.

4. The method according to claim 3, characterized in that The method further comprises: The threat level of the threat indicator IOC is determined based on the first data source.

5. The method according to claim 3, characterized in that: The step of aggregating the data in the first data source and the data in the second data source based on the data in the first data source to obtain comprehensive data includes: Based on the data of the first data source, data in the first data source and the second data source are integrated, supplemented and conflict resolved to obtain the comprehensive data.

6. The method according to any one of claims 2 to 5, characterized in that: The first data source and the second data source are data sources that have passed whitelist verification.

7. The method according to any one of claims 2 to 5, characterized in that: If there is data update in the first data source, the method further includes: Obtaining updated third statistical information and a third confidence coefficient of the first data source; calculating a third score set of the first data source and a fourth score set of the second data source according to the second statistical information, the second confidence coefficient, the third statistical information and the third confidence coefficient, wherein the third score set includes at least two of the following scores: a third contribution score, a third matching score, a third applicability score, a third richness score, a third difference score and a third timeliness score, and the fourth score set includes at least two of the following scores: a fourth contribution score, a fourth matching score, a fourth applicability score, a fourth richness score, a fourth difference score and a fourth timeliness score; the number of scores included in the third score set and the performance indicators involved are the same as those of the fourth score set; A third evaluation result of the first data source is calculated based on the third score set, and a fourth evaluation result of the second data source is calculated based on the fourth score set, the third evaluation result is used to characterize the data quality of the updated first data source, and the fourth evaluation result is used to evaluate the data quality of the second data source.

8. The method according to any one of claims 2 to 5, characterized in that: The calculating a first evaluation result of the first data source based on the first score set includes: Calculate the average of the scores in the first score set, and record it as the first evaluation result; The calculating a second evaluation result of the second data source based on the second score set includes: An average value of the scores in the second score set is calculated and recorded as the second evaluation result.

9. The method according to claim 1, characterized in that: The data sources include a first data source, a second data source, and a third data source, and obtaining statistical information and a confidence coefficient of each data source includes: Obtaining first statistical information and a first confidence coefficient of the first data source, obtaining second statistical information and a second confidence coefficient of the second data source, and obtaining fourth statistical information and a fourth confidence coefficient of the third data source; The calculating, according to the statistical information and confidence coefficient of each data source, a score set of each data source in the data sources includes: According to the first statistical information, the first confidence coefficient, the second statistical information, the second confidence coefficient, the fourth statistical information and the fourth confidence coefficient, a first score set for the first data source is calculated, a second score set for the second data source is calculated, and a fifth score set for the third data source is calculated, wherein the first score set includes at least two of the following scores: a first contribution score, a first matching score, a first applicability score, a first richness score, a first difference score and a first timeliness score, the second score set includes at least two of the following scores: a second contribution score, a second matching score, a second applicability score, a second richness score, a second difference score and a second timeliness score, and the fifth score set includes at least two of the following scores: a fifth contribution score, a fifth matching score, a fifth applicability score, a fifth richness score, a fifth difference score and a fifth timeliness score, and the first score set, the second score set and the fifth score set have the same number of scores and the same performance indicators involved; The step of calculating, for each of the data sources, an evaluation result of the data source based on the score set of the data source, includes: A first evaluation result of the first data source is calculated based on the first score set, a second evaluation result of the second data source is calculated based on the second score set, and a fifth evaluation result of the third data source is calculated based on the fifth score set, wherein the first evaluation result is used to characterize data quality of the first data source, the second evaluation result is used to evaluate data quality of the second data source, and the fifth evaluation result is used to characterize data quality of the third data source.

10. The method according to claim 9, characterized in that The method further comprises: According to the first evaluation result, the second evaluation result, and the fifth evaluation result, determining that the data quality of the first data source is higher than the data quality of the second data source, and the data quality of the second data source is higher than the data quality of the third data source; Based on the data of the first data source, the data in the first data source, the data in the second data source and the data in the third data source are aggregated to obtain comprehensive data, and the comprehensive data is used for network security analysis. During the aggregation process, the aggregation operation is performed on the data in multiple data sources in order from high to low data quality indicated by the evaluation results.

11. A data source quality assessment device, characterized in that: include: A first acquisition unit, used to acquire statistical information and confidence coefficients of each data source; a first calculation unit, configured to calculate a score set of each of the data sources according to the statistical information and confidence coefficient of each of the data sources, the score set including at least two of the following scores: a contribution score, a matching score, a suitability score, a richness score, a difference score, and a timeliness score, and the number of scores and the performance indicators involved in the score set of each of the data sources are the same; a second calculation unit, configured to calculate, for each of the data sources, an evaluation result of the data source based on the score set of the data source, wherein the evaluation result of each data source is used to characterize the data quality of the data source; Among them, the contribution score refers to the proportion of data pushed by the data source to the total amount of data pushed by all data sources; the matching score refers to the proportion of data of any data type pushed by the data source to the total amount of data of this data type pushed by all data sources; the applicability score refers to the proportion of data corresponding to any data type that has passed the evaluation query and hit in the data source to the total amount of data of this data type pushed by the data source; the richness score refers to the richness of auxiliary information in each data source, and the auxiliary information includes one or more of secure socket protocol certificates, passive domain name resolution books, and digital signatures; the difference score refers to the overlap ratio of data between data sources; the timeliness score refers to the data update frequency and the proportion of first-released data results of each data source.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 10 according to the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 10.

Images