A threat intelligence quality crowd-sourcing evaluation method based on a reputation mechanism

Abstract

The application provides a threat intelligence quality crowdsourcing evaluation method based on a reputation mechanism, and relates to the field of computer network security.The method provided by the application comprises the following steps: generating an objective score based on objective dimension evaluation by an automatic detection technology; obtaining an initial reputation value based on an additional reputation value and an initial reputation contribution value, and obtaining an initial subjective score of a subjective dimension by a crowdsourcing evaluator; screening out non-anomalous scores based on a weighted absolute median deviation, and obtaining a subjective score and a score entropy by post-hoc aggregation of the non-anomalous scores based on a weighted Dirichlet distribution model; obtaining a score deviation based on the initial subjective score and the subjective score, and constructing a reputation function to update the initial reputation value to obtain a reputation value; constructing a reward function based on the score deviation and the reputation value, and distributing rewards to the crowdsourcing evaluators; and obtaining a quality score by weighted fusion of each score.The application effectively realizes quality evaluation of threat intelligence by combining a crowdsourcing mechanism and a reputation reward mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer network security, and in particular to a threat intelligence quality crowdsourcing evaluation method based on a reputation mechanism. BACKGROUND

[0002] With the continuous evolution of network attack means, especially the frequent occurrence of complex attack patterns such as APT (Advanced Persistent Threat), ransomware, supply chain attack and zero-day vulnerability, network space security is facing severe challenges, and the traditional defense system is often difficult to effectively respond. Under this background, cyber threat intelligence (CTI) as a key knowledge resource to understand the intentions and means of attackers has become the core support of the modern network security system. High-quality threat intelligence can enable attack early warning, strategy formulation, joint defense and rapid response, and identify potential threats before attacks occur, which is the key to the transition from "passive defense" to "active defense". However, the quality of intelligence seriously restricts the actual application value: on the one hand, the intelligence content may be false, false, invalid or lack of context explanation, making it difficult to judge the credibility; on the other hand, the intelligence format lacks unified standards and inconsistent semantic structures, hindering effective integration across platforms. The superposition of these problems may lead to incorrect security policy configuration, alarm flooding and even false positives, which may even pose risks to the organization's security operations.

[0003] Current intelligence quality evaluation methods mainly rely on two methods: one is expert manual review and scoring, and the other is automatic scoring based on pre-set rules or models. However, manual evaluation is slow in response, high in personnel cost, and difficult to cover the growing number of intelligence; automatic evaluation is difficult to understand the context semantics, behavior chain association and potential impact of intelligence. Therefore, it is necessary to provide a solution to improve the above problems. SUMMARY

[0004] The purpose of the present application is to provide a threat intelligence quality crowdsourcing evaluation method based on a reputation mechanism, which can improve the problems of slow response, high cost, low coverage and model singularity in existing intelligence quality evaluation methods.

[0005] The threat intelligence quality crowdsourcing evaluation method based on the reputation mechanism provided by the present application comprises:

[0006] generating an objective score based on the objective dimension of the threat intelligence evaluated by the automatic detection technology;

[0007] obtaining an initial reputation value based on the additional reputation value and the initial reputation contribution value, and obtaining an initial subjective score of the subjective dimension of the threat intelligence by the crowdsourcing evaluator based on the initial reputation value;

[0008] screening non-abnormal scores in the initial subjective scores based on a weighted absolute median deviation, performing posterior aggregation on the non-abnormal scores based on a weighted Dirichlet distribution model to obtain subjective scores and score entropy;

[0009] obtaining score deviation based on the initial subjective scores and the subjective scores, constructing a reputation function based on the score deviation and the score entropy, updating the initial reputation value based on the reputation function to obtain a reputation value;

[0010] constructing a reward function based on the score deviation and the reputation value, and performing reward distribution on the crowd assessors based on the reward function;

[0011] weighting and fusing the objective scores and the subjective scores to obtain a quality score of threat intelligence.

[0012] The application provides a reputation mechanism-based threat intelligence quality crowd assessment method, which realizes assessment of threat intelligence quality through automatic detection technology and crowd assessment mechanism, and improves the quality of crowd assessment based on a reputation incentive mechanism.

[0013] Optionally, the automatic detection technology includes preset rules, pattern matching and statistical analysis methods.

[0014] Optionally, the objective dimensions include syntax accuracy, pattern integrity, expression conciseness and logic consistency; and the subjective dimensions include content accuracy, attack context integrity, operability, relevance, source credibility and timeliness.

[0015] Optionally, the initial reputation value is obtained based on an additional reputation value and an initial reputation contribution value, and the initial subjective scores of the subjective dimensions of the threat intelligence by the crowd assessors are obtained based on the initial reputation value, including: testing the crowd assessors who register for the first time or participate in the scoring task to obtain test results, quantifying the test results to obtain scores, and mapping the scores to the initial reputation contribution value through a linear function or a segmented function; if the crowd assessors hold authentication information, providing corresponding additional reputation values according to different authentication information; linearly combining the additional reputation values and the initial reputation contribution values to obtain the initial reputation value, and obtaining the initial subjective scores of the subjective dimensions of the threat intelligence by the crowd assessors based on the initial reputation value.

[0016] Optionally, the non-abnormal scores in the initial subjective scores are screened based on a weighted absolute median deviation, including:

[0017] defining a score set, the score set including the initial subjective scores of the crowd assessors and the reputation values of the crowd assessors; and calculating the weight of each crowd assessor based on the reputation value;

[0018] arranging initial subjective scores in a score set in ascending order, summing all weights to obtain a total weight, determining a median position based on the total weight to obtain a weighted median;

[0019] dividing the score set into a left set and a right set based on the weighted median, calculating left deviation sets and right deviation sets by respectively calculating deviations of the left set and the right set from the weighted median, and respectively calculating weighted absolute median deviations of the left deviation sets and the right deviation sets based on weights and performing smoothing processing thereon;

[0020] respectively calculating tolerance deviation thresholds of the left set and the right set based on the weighted median deviations, performing anomaly detection on the initial subjective scores based on the tolerance deviation thresholds, and retaining non-anomalous scores.

[0021] Optionally, when performing posterior aggregation on the non-anomalous scores based on a weighted Dirichlet distribution model to obtain subjective scores and score entropy, the method comprises the following steps: defining an effective sample number based on weights, defining a normalized weight based on the effective sample number, defining posterior parameters based on the normalized weight for score levels, performing posterior aggregation on the non-anomalous scores based on the posterior parameters to obtain the subjective scores and the score entropy; the score levels comprise integer levels of 1, 2, 3, 4 and 5.

[0022] Optionally, when constructing a reputation function, the method comprises the following steps: calculating a score deviation of each crowd-sourced evaluator between an initial subjective score and the subjective score to obtain the score deviation, constructing the reputation function based on the score deviation and the score entropy, updating the initial reputation value within a limited range based on the reputation function to obtain a reputation value, and completing evaluation of a subjective dimension of threat intelligence in the next round based on the reputation value.

[0023] Optionally, when constructing a reward function, the method comprises the following steps: constructing the reward function based on the score deviation and the reputation value, and performing weighted allocation of a budget value of the subjective dimension to be evaluated to the crowd-sourced evaluators.

[0024] Optionally, when performing weighted fusion on the objective scores and the subjective scores to obtain a quality score of threat intelligence, the method comprises the following steps: performing weighted summation on the objective scores and the subjective scores based on a first preset weight allocated to the objective dimension and a second preset weight allocated to the subjective dimension to obtain the quality score of threat intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A flowchart of a threat intelligence quality crowd-sourced evaluation method based on a reputation mechanism is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the common meanings thereof by those skilled in the art.

[0027] Referring to Figure 1 The present application provides a threat intelligence quality crowd-sourcing evaluation method based on a reputation mechanism, comprising the following steps:

[0028] S1, generating an objective score based on an objective dimension of threat intelligence evaluated by an automated detection technology;

[0029] S2, obtaining an initial reputation value based on an additional reputation value and an initial reputation contribution value, and obtaining an initial subjective score of the subjective dimension of threat intelligence by a crowd-sourcing evaluator based on the initial reputation value;

[0030] S3, screening out non-anomalous scores in the initial subjective score based on a weighted absolute median deviation, and performing posterior aggregation on the non-anomalous scores based on a weighted Dirichlet distribution model to obtain a subjective score and a score entropy;

[0031] S4, obtaining a score deviation based on the initial subjective score and the subjective score, constructing a reputation function based on the score deviation and the score entropy, and updating the initial reputation value based on the reputation function to obtain a reputation value;

[0032] S5, constructing a reward function based on the score deviation and the reputation value, and distributing rewards to the crowd-sourcing evaluators based on the reward function;

[0033] S6, obtaining a quality score of threat intelligence by weightedly fusing the objective score and the subjective score.

[0034] In fact, the crowd-sourcing evaluation method provided by the present application can complete the evaluation of the objective dimension and the subjective dimension of threat intelligence respectively through the automated detection technology and the crowd-sourcing evaluation mechanism, so as to achieve the purposes of high response, low cost, high coverage and diversity of intelligence quality evaluation. Moreover, the reputation reward mechanism is constructed, so as to optimize the crowd-sourcing evaluation of the subjective dimension and achieve the purposes of stability of the crowd-sourcing evaluation mechanism and high-quality evaluation.

[0035] The present application is directed to structured threat information expression (STIX), and a three-dimensional evaluation framework covering structure layer, information layer and value layer is constructed. The framework comprehensively considers the structural integrity, information authenticity and practical application value of threat information, and aims to provide a comprehensive, detailed and configurable quality evaluation framework. The inherent hierarchical structure and rich semantic elements of the STIX standard provide a natural advantage for the construction of multi-dimensional quality evaluation indicators of the framework.

[0036] Specifically, the structure layer (Structure Layer) focuses on the formal normativity and structural integrity of threat information, and the evaluation mainly depends on objective and automated rule checking. The evaluation dimensions include:

[0037] Syntax accuracy: check whether the intelligence data conforms to the syntax specification and mode definition of STIX.

[0038] Mode integrity: assess the filling of mandatory fields and identify missing key information.

[0039] Expressiveness: measure the redundancy of intelligence content to ensure clear and unambiguous information.

[0040] Specifically, the information layer (Information Layer) goes deep into the intelligence content and semantic level, and some evaluation dimensions need to be combined with automated tools, and more need to introduce crowdsourcing mechanism for subjective judgment. The evaluation dimensions include:

[0041] Content accuracy: verify whether the entities, events and relationships described by the intelligence are real and correct.

[0042] Attack context completeness: evaluate the completeness of the description of key elements of attack behavior (such as attack chain, exploitation tool, affected asset type, attack phase and attacker intention).

[0043] Logical consistency: check whether the association, ownership and timing logic of objects within the intelligence are consistent, and identify structural problems such as "broken chain" or uncited isolated objects.

[0044] Specifically, the value layer (Value Layer) evaluates the practicality and influence of threat information, reflecting its guiding significance for actual security decision-making and action, and highly depends on the professional knowledge and experience of crowdsourcing participants. The evaluation dimensions include:

[0045] Operability: whether the intelligence can be directly converted into defense measures, detection rules or response actions.

[0046] Relevance: whether the intelligence content is related to the organization corresponding to the intelligence label uploaded by the intelligence submitter.

[0047] Source Credibility: Assess the reliability of third-party sources (e.g. mainstream manufacturers, authoritative organizations) cited in the intelligence.

[0048] Timeliness: Whether the intelligence has current value, is not outdated or invalid.

[0049] In some embodiments, when the objective score is generated based on the evaluation of the objective dimensions of threat intelligence in step S1 using automated detection technology, the objective dimensions can be grammatical accuracy, pattern completeness, concise expression, and logical consistency. The system will automatically generate an objective score for the objective dimensions according to pre-set rules, pattern matching or statistical analysis methods.

[0050] Specifically, grammatical accuracy is evaluated using the stix2.validator tool, and the evaluation formula is as follows:

[0051] ;

[0052] Wherein, is the scoring rule for grammatical accuracy.

[0053] Specifically, the scoring formula for pattern completeness is as follows:

[0054] ;

[0055] ;

[0056] Wherein, is the completeness ratio; is the number of recommended fields actually filled in; is the theoretical total number of recommended field sets, i.e. Wherein, is the recommended field set (such as external_references, labels, description, object_marking_refs, etc.); is the scoring rule for pattern completeness.

[0057] Specifically, the scoring formula for concise expression is as follows:

[0058] ;

[0059] ;

[0060] ;

[0061] Wherein, is the empty field rate, is the number of empty fields (such as description = ""), For all field numbers, For normalized nesting depth, For maximum JSON nesting depth (used for recursive or traversal structure calculation), For expressing indirectness scoring rules.

[0062] Specifically, the scoring formula of logical consistency is as follows:

[0063]

[0064]

[0065] wherein, is a consistency coefficient; is the number of invalid references (such as non-existent IDs, circular references, etc.); is the number of isolated nodes (no references and not referenced); is the total number of nodes in the graph, that is, is the scoring rule of logical consistency.

[0066] In some embodiments, the subjective dimensions in step S2 can be content accuracy, attack context integrity, operability, relevance, source credibility, and timeliness. The crowd-sourcing evaluators score each subjective dimension, and the score level can be set as an integer level of 1, 2, 3, 4, and 5 (1 represents the lowest quality and 5 represents the highest quality). To effectively suppress the interference of “noise participants” and malicious behaviors on the score results, the present application adopts a dynamic weighted aggregation mechanism based on reputation value (Reputation). The score of each evaluator will be weighted by its current reputation value when aggregating finally. The higher the reputation value of the evaluator, the greater the weight of its score in aggregation. This reputation value is not only used as a weighting coefficient when aggregating subjective scores, but also directly determines the overall influence of the evaluator in the crowd-sourcing task, the degree of final score adoption, and the incentive benefit distribution. To improve the accuracy and stability of subjective scores, the present application designs a dynamic evaluator reputation value updating mechanism, aiming to suppress speculative scoring, extreme scoring, and other irrational behaviors, while giving positive incentives to evaluators with stable scores close to the group consensus. In this mechanism, each evaluator participating in the crowd-sourcing scoring task will be assigned a dynamically changing reputation value (a positive real number). Given the central role of reputation value in the mechanism, it is of critical importance to set its initial value reasonably.

[0067] ​​​In some embodiments, the initial reputation value is obtained in step S2 based on an additional reputation value and an initial reputation contribution value, and when obtaining the initial subjective score of the subjective dimension of threat intelligence by the crowd-sourcing assessor based on the initial reputation value, the crowd-sourcing assessor needs to complete a set of standardized tests before first registration or participation in the scoring task, covering structured threat information (STIX), attack context analysis ability, and basic knowledge of network security terminology. The test results will be quantified as a score s quiz ∈[0,100] The score will be mapped to the initial reputation contribution value by a linear function or a segmented function If the crowd-sourcing assessor holds recognized certification information in the network security field (such as CISSP, CEH, GCTI, CISA, etc.), the platform can be bound, and the platform will automatically verify or manually audit the validity and authority of the bound certification information through the API interface, and set a predefined weighting coefficient for the crowd-sourcing assessor according to the professional degree and coverage range of different certification information (wherein, represents the first certificate). After the certification information is verified, the system will calculate and grant the corresponding additional reputation value to the crowd-sourcing assessor. The initial reputation value of the crowd-sourcing assessor is obtained by linearly combining the initial reputation contribution value and the additional reputation value. When obtaining the initial subjective score of the subjective dimension of threat intelligence by the crowd-sourcing assessor, the initial reputation value is weighted.

[0068] Specifically, the calculation formula of the additional reputation value is as follows:

[0069] ;

[0070] wherein, is the validity of the first certificate, represents that the first certificate has been verified to be valid, represents that the first certificate has been verified to be invalid.

[0071] Specifically, the calculation formula of the initial reputation value is as follows:

[0072] ;

[0073] wherein, the function represents limiting the value within a set range, and This sets the upper and lower limits for the initial reputation score. This setting ensures that crowdsourced evaluators with professional qualifications are given higher weight in the early stages of the platform, which helps to improve the credibility and stability of the initial crowdsourced scores, while avoiding excessive interference from newly joined crowdsourced evaluators in the overall scoring system.

[0074] In some embodiments, to address the issues of quality fluctuations, extreme value interference, and large differences in opinions among evaluators in subjective dimension ratings in step S3, this invention proposes a two-stage crowdsourcing aggregation mechanism: First, a direction-aware weighted absolute median deviation (MAD) method is used to remove extreme ratings, improving the purity of the rating data before aggregation; then, based on Bayesian inference, a weighted Dirichlet distribution model is used to perform posterior fusion of the crowdsourcing results, outputting the expected value of the quality rating (subjective rating) and the rating consistency (rating entropy), enhancing the expressiveness and credibility of the subjective ratings. This mechanism is suitable for multi-evaluator rating fusion scenarios under subjective scales (such as Likert 1–5) and has good robustness and scalability.

[0075] In some embodiments, when filtering out non-abnormal scores from the initial subjective scores based on the weighted absolute median deviation in step S3, a score set is defined, which includes the initial subjective scores of the crowdsourced evaluators and their reputation scores. The weight of each crowdsourced evaluator is calculated based on their reputation scores. The initial subjective scores in the score set are sorted in ascending order, and all weights are summed to obtain the total weight. The median position is determined based on the total weight to obtain the weighted median. Based on the weighted median, the score set is divided into a left set and a right set. The deviations of the left set and the right set from the weighted median are calculated to obtain the left deviation set and the right deviation set, respectively. The weighted absolute median deviation of the left deviation set and the right deviation set are calculated based on their weights and then smoothed. The tolerance deviation thresholds for the left set and the right set are calculated based on the weighted median deviations. Anomaly detection is performed on the initial subjective scores based on the tolerance deviation thresholds, and non-abnormal scores are retained. The specific steps are as follows:

[0076] S301, Let the rating set be... ,in, For crowdsourced evaluators Initial subjective ratings, , The reputation score of the crowdsourcing evaluator. The total number of samples, each sample consists of... and Composition; weights for each sample are calculated based on reputation scores. ,Right now ;

[0077] S302, Transfer the rating set ascending order, obtaining an ordered sequence wherein, is the smallest value, is the total number of scored samples;

[0078] summing all weights to obtain total weight , i.e. ; finding the smallest index satisfying , determining the median position; outputting the weighted median, the weighted median being ;

[0079] S303, according to the weighted median , the score set is divided into a left set and a right set ;

[0080] respectively calculating the deviation of the left and right sets from the weighted median to obtain a left deviation set and a right deviation set ;

[0081] based on the weight, respectively calculating the weighted absolute median deviation of the left deviation set and the weighted absolute median deviation of the right deviation set , wherein, is the median calculation considering the weight ; to prevent the deviation from being zero under discrete scores, the result is smoothed:

[0082] ;

[0083] ;

[0084] S304, defining the tolerance deviation threshold of each crowd evaluator based on the weighted median deviation, the formula being as follows:

[0085] ;

[0086] ;

[0087] wherein, and are the tolerance deviation threshold of the left set and the tolerance deviation threshold of the right set respectively, is an amplification factor of the absolute median deviation, is a reputation adjustment coefficient;

[0088] S305, based on the tolerance deviation threshold, each​ Anomaly detection is performed, and non-anomalous scores are reserved for subsequent aggregation, and the anomaly detection rule is as follows:

[0089] .

[0090] In some embodiments, in step S3, the non-anomalous scores are post-processed based on the weighted Dirichlet distribution model, and when obtaining the subjective score and the score entropy, the effective sample number is defined based on the weight, the normalized weight is defined based on the effective sample number, the posterior parameter is defined for the score grade based on the normalized weight, the non-anomalous scores are post-processed based on the posterior parameter, and the subjective score and the score entropy are obtained. The specific steps are as follows:

[0091] S311, in order to avoid the dominance of a small number of high-reputation evaluators on the score, an effective sample number adjustment factor is introduced, and the weight set is , the effective sample number is defined as:

[0092] ;

[0093] The weight is normalized to obtain the normalized weight , which keeps the weight proportion unchanged and constrains the total weight to an information equivalent sample number, improving the fairness and stability of aggregation; the formula used is as follows:

[0094] ;

[0095] wherein, is the number of weights; is the summation symbol subscript, that is, ;

[0096] S312, for each score grade , the posterior parameter thereof is defined based on the normalized weight , and the formula used is as follows:

[0097] ;

[0098] wherein, is the score prior (the default setting is 1.0 for smoothing processing), is an indicator function, and its specific meaning is:

[0099] ;

[0100] S313, the non-anomalous scores are post-processed based on the posterior parameter, and the subjective score E[ S ] and the score entropy Wherein, the score entropy is used for measuring the evaluation consistency and subsequent risk labeling or as the basis for manual compliance, and the greater the entropy value represents the greater the score difference; the formula used is as follows:

[0101] ;

[0102] .

[0103] In some embodiments, in step S4 in each round of evaluation task, the system first calculates the deviation degree between the initial subjective score of each evaluator and the final aggregated score, to obtain the score deviation , wherein, is the final aggregated score (the value takes the subjective score E[ S ] ); to judge the consistency degree of the current group score, the score entropy is introduced to construct the reputation function; to prevent the reputation value from fluctuating too fast or being manipulated by extreme behavior, a limiting function is added when updating the reputation value. The updated reputation value is used to complete the next round of evaluation task. This mechanism has significant adaptive adjustment and positive guidance characteristics in design, and its core logic is embodied as follows: by comprehensively considering the score deviation degree of the evaluator and the group score consensus degree (quantified by the score entropy index), the response strength of individual score behavior is dynamically adjusted; when the evaluator's score deviates slightly from the group consensus, the system gives positive incentives (reputation value is increased) to encourage it to continue to provide reasonable judgments; when the group score opinions are greatly different (the consensus degree is low), the system moderately increases the tolerance of individual deviation to avoid suppressing a few evaluators who may have independent perspectives; in the scenario where the group score is highly consistent (the consensus degree is high and the score entropy is low), if the evaluator shows significantly deviated malicious or arbitrary score behavior, the system will impose severe punishment (the reputation value is greatly reduced) to effectively suppress interference; the mechanism allows evaluators whose reputation value is damaged due to deviation or error to gradually restore the reputation value through stable and reasonable score performance in subsequent multiple rounds of tasks, forming a virtuous closed loop of "incentive-feedback-repair". In summary, the dynamic reputation value updating mechanism proposed in the application significantly improves the accuracy and robustness of the aggregated result of subjective score by continuously and intelligently regulating the behavior of evaluators, providing a core guarantee for the reliable and stable operation of the threat intelligence quality evaluation system.

[0104] Specifically, the reputation function is used to adjust the reputation value of the crowd-sourced evaluator, and the function used is as follows:

[0105] ;

[0106] Wherein, is the reputation value adjustment amount, is the tolerance deviation threshold, This is a positive reward coefficient (e.g., 0.05). This is a negative penalty coefficient (e.g., 0.1). The maximum value of the rating entropy ( (This is the theoretical maximum value).

[0107] Specifically, the reputation score update formula is as follows:

[0108] ;

[0109] in, For the updated reputation value, The lower limit of the reputation score ( ), For crowdsourcing evaluators in the first Reputation score in the round of evaluation tasks This is the upper limit for the change in reputation value in a single round (e.g., 0.5).

[0110] In some embodiments, to incentivize evaluators to provide accurate and rational scores in step S5, while simultaneously improving the system's robustness and participation, this invention designs a differentiated reward mechanism based on scoring bias and reputation value. A reward function is constructed based on scoring bias and reputation value, allocating a specific dimension to be evaluated from the intelligence submitter. Budget A weighted reward system is applied to crowdsourced evaluators who effectively participate in the assessment. This reward mechanism deeply integrates a scoring deviation penalty mechanism with a reputation value bonus effect, achieving precise incentives for high-quality evaluation behavior. Simultaneously, a relatively normalized allocation strategy is adopted to ensure the fairness of rewards and the sustainability of the system while strictly adhering to pre-set budget constraints. Ultimately, a positive feedback incentive ecosystem is constructed that can continuously optimize evaluator behavior and improve intelligence quality.

[0111] Specifically, crowdsourced evaluators In dimensions Rewards obtained The calculation formula is as follows:

[0112] ;

[0113] in, For crowdsourced evaluators In dimensions Scoring bias; The preset maximum tolerance deviation (e.g., set to 2 points); In dimension The normalization factor is used to ensure the conservation of the total budget. As the crowdsourced evaluator in the current round Reputation value.

[0114] wherein, are defined as follows:

[0115] ;

[0116] wherein, is a set of assessors who submitted valid scores on dimension , is a reputation value of the crowd-sourced assessor on dimension , is a reputation value of the crowd-sourced assessor in the current round.

[0117] In some embodiments, the quality score of the threat intelligence in step S6 is calculated as follows: E [ S info ] The score will be obtained by weighted combination of the scores of each level and each dimension, which integrates the score results of objective and subjective dimensions, and will be used as an important basis for subsequent intelligence screening, traceability analysis, intelligence provider incentive and security policy making, so as to effectively improve the utilization efficiency and defense effect of threat intelligence. The calculation formula is as follows:

[0118] ;

[0119] wherein, is a preset weight of dimension , which can be flexibly configured and adjusted according to actual application scenarios and security requirements to adapt to the focus of different organizations on intelligence quality; is the total number of evaluation dimensions; E [ S d ] is a representative sub-score aggregated from dimension .

[0120] Although the embodiments of the present application have been described in detail above, it is obvious to those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes all fall within the scope and spirit of the present application described in the claims. Moreover, the present application described herein can have other embodiments and can be implemented or realized in various ways.

Claims

1. A crowdsourced assessment method for threat intelligence quality based on a reputation mechanism, characterized in that, include: An objective score is generated based on the objective dimensions of threat intelligence assessment using automated detection technology. These objective dimensions include grammatical accuracy, pattern completeness, conciseness of expression, and logical consistency. An initial reputation value is obtained based on additional reputation value and initial reputation contribution value. Based on this initial reputation value, an initial subjective score is obtained from the crowdsourced evaluators' subjective dimensions of threat intelligence. This includes: testing crowdsourced evaluators who register for the first time or participate in the scoring task to obtain test results; quantifying the test results to obtain a score; and mapping the score to the initial reputation contribution value using a linear function or piecewise function. If the crowdsourced evaluator holds authentication information, corresponding additional reputation value is provided based on different authentication information. The subjective dimensions include content accuracy, attack context integrity, operability, relevance, source credibility, and timeliness. Non-abnormal scores in the initial subjective scores are selected based on the weighted absolute median deviation, and the non-abnormal scores are aggregated posteriorly based on the weighted Dirichlet distribution model to obtain the subjective scores and score entropy. The process includes: calculating the degree of deviation between the initial subjective rating and the subjective rating for each crowdsourced evaluator to obtain the rating deviation; constructing a reputation function based on the rating deviation and the rating entropy; and updating the initial reputation value based on the reputation function to obtain a reputation value. A reward function is constructed based on the scoring deviation and the reputation value, and the budget value of the subjective dimension to be evaluated is weighted and allocated to the crowdsourcing evaluators; rewards are then allocated to the crowdsourcing evaluators based on the reward function. The quality score of threat intelligence is obtained by weighted fusion of the objective score and the subjective score.

2. The crowdsourcing evaluation method as described in claim 1, characterized in that, The automated detection technology includes preset rules, pattern matching, and statistical analysis methods.

3. The crowdsourcing evaluation method as described in claim 1, characterized in that, When obtaining an initial reputation value based on additional reputation value and initial reputation contribution value, and obtaining an initial subjective score of the crowdsourced evaluator's subjective dimension of threat intelligence based on the initial reputation value, the process includes: linearly merging the additional reputation value and the initial reputation contribution value to obtain the initial reputation value, and obtaining an initial subjective score of the crowdsourced evaluator's subjective dimension of threat intelligence based on the crowdsourced evaluator's initial reputation value.

4. The crowdsourcing evaluation method as described in claim 1, characterized in that, When filtering out non-abnormal scores from the initial subjective scores based on the weighted absolute median deviation, the following steps are included: Define a rating set, which includes the initial subjective ratings of crowdsourced evaluators and the reputation scores of crowdsourced evaluators; calculate the weight of each crowdsourced evaluator based on the reputation scores; The initial subjective ratings in the rating set are sorted in ascending order, and all weights are summed to obtain the total weight. The median position is determined based on the total weight to obtain the weighted median. The score set is divided into a left set and a right set based on the weighted median. The deviations of the left set and the right set from the weighted median are calculated to obtain the left deviation set and the right deviation set. The weighted absolute median deviation of the left deviation set and the right deviation set is calculated based on the weights and then smoothed. Based on the weighted median deviation, tolerance deviation thresholds are calculated for the left and right sets respectively. Anomaly detection is performed on the initial subjective scores based on the tolerance deviation thresholds, and non-abnormal scores are retained.

5. The crowdsourcing evaluation method as described in claim 3, characterized in that, When performing posterior aggregation of non-abnormal ratings based on a weighted Dirichlet distribution model to obtain subjective ratings and rating entropy, the process includes: defining the number of valid samples based on weights, defining normalized weights based on the number of valid samples, defining posterior parameters for rating levels based on normalized weights, and performing posterior aggregation of non-abnormal ratings based on the posterior parameters to obtain subjective ratings and rating entropy; the rating levels include integer levels of 1, 2, 3, 4, and 5.

6. The crowdsourcing evaluation method as described in claim 1, characterized in that, When constructing the reputation function, the process includes: updating the initial reputation value within a limited range based on the reputation function to obtain a reputation value; and completing the assessment of the subjective dimension of the next round of threat intelligence based on the reputation value.

7. The crowdsourcing evaluation method as described in claim 1, characterized in that, When obtaining a quality score for threat intelligence by weighted fusion of the objective score and the subjective score, the process includes: weighting and summing the objective score and the subjective score based on a first preset weight assigned to the objective dimension and a second preset weight assigned to the subjective dimension to obtain a quality score for threat intelligence.

Images