A false threat intelligence identification model robustness quantitative analysis method and device

Abstract

The present application relates to the field of information technology, and discloses a false threat intelligence identification model robustness quantitative analysis method and device.The method comprises extracting source identifiers, propagation path node sequences and message content texts from intelligence messages, and constructing a basic data set for robustness quantitative analysis; modifying the data in the basic data set, and calculating the performance index change of the identification model; based on the historical intelligence message set and the performance index change, calculating a comprehensive robustness index through a weighted fusion algorithm; based on the performance index change and the comprehensive robustness index, generating a defect label set; and based on the defect label set, dynamically adjusting the identification model parameters through reinforcement learning.The present application aims to comprehensively quantify multiple performance indexes to analyze the robustness of the identification model, locate model defects, and optimize the model accordingly, thereby improving the anti-interference performance of the identification model in a complex network environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a false threat intelligence identification model robustness quantitative analysis method and device. BACKGROUND

[0002] With the increasing complexity and organization of network attack means, false threat intelligence usually adopts the means of forged source identifier, modified propagation path and injected text against samples to bypass detection, which makes the current false threat intelligence identification model face serious robustness challenges.

[0003] In the prior art, the robustness analysis method of the identification model is mainly realized by relying on a single performance indicator. However, the analysis of a single performance indicator is difficult to comprehensively and systematically quantify the real robustness level of the identification model when facing different attack types. More importantly, the existing analysis method is difficult to accurately locate the specific root cause of the identification model robustness not meeting the standard, and lacks a targeted model parameter optimization mechanism based on defect positioning. These limitations make it difficult for the identification model to effectively cope with the dynamic evolution and increasingly complex attack means of the network environment.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] In view of the problem in the prior art that the robustness analysis method of the identification model only relies on a single performance indicator and lacks a defect positioning mechanism, the present application provides a false threat intelligence identification model robustness quantitative analysis method and device, which aims to comprehensively quantify multiple performance indicators to analyze the robustness of the identification model, locate the model defects, and optimize the model accordingly, thereby improving the anti-interference performance of the identification model in a complex network environment.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] A false threat intelligence identification model robustness quantitative analysis method, comprising:

[0008] extracting a source identifier, a propagation path node sequence and a message content text from the intelligence message to construct a basic data set for robustness quantitative analysis; the source identifier includes identity identification, authentication information and credibility level of the intelligence publishing subject;

[0009] modifying the data in the basic data set and calculating the performance indicator change of the identification model;

[0010] based on the historical intelligence message set and the performance indicator change, calculating a comprehensive robustness indicator by a weighted fusion algorithm;

[0011] generating a defect label set based on the performance index change and the comprehensive robustness index;

[0012] Based on the defect label set, the model parameter is dynamically adjusted by reinforcement learning.

[0013] As a preferred scheme of the present application, the modification of the data in the basic data set and the calculation of the performance index change of the identification model include:

[0014] Modifying the source identifier in the basic data set, including replacing the source identifier, or deleting the source identifier, or forging the source identifier, calculating the output accuracy rate change rate S1 of the identification model;

[0015] Modifying the nodes in the propagation path node sequence in the basic data set, calculating the F1 value fluctuation S2 of the identification model;

[0016] Injecting text adversarial samples into the message content text in the basic data set, calculating the recall rate deviation S3 of the identification model.

[0017] As a preferred scheme of the present application, the modification of the nodes in the propagation path node sequence in the basic data set includes:

[0018] Random forest importance sampling is used to select the deleted nodes; or, a pseudo node is added and the interaction timestamp is forged.

[0019] As a preferred scheme of the present application, the injection of text adversarial samples into the message content text in the basic data set includes:

[0020] The synonyms in the message content text are replaced by a pre-trained language model; or,

[0021] Adjusting the syntax of the message content text; or,

[0022] Inserting interference sentences into the message content text.

[0023] As a preferred scheme of the present application, the calculation of the comprehensive robustness index based on the historical intelligence message set and the performance index change by the weighted fusion algorithm includes:

[0024] Obtaining the historical intelligence message set, including the false threat intelligence samples, the real threat intelligence samples and the corresponding identification model robustness evaluation results within a predetermined time;

[0025] Based on the historical intelligence message set, a weighted linear regression model is used to calculate the weight coefficient, including the source modification weight W1, the propagation path modification weight W2 and the content modification weight W3;

[0026] Based on the weight coefficient, the comprehensive robustness index T is calculated by the weighted fusion algorithm, and the calculation formula is:

[0027] T = W1 x S1 + W2 x S2 + W3 x S3

[0028] Wherein, W1 + W2 + W3 = 1.

[0029] As a preferred scheme of the present application, the generation of the defect label set based on the performance index change and the comprehensive robustness index comprises:

[0030] Comparing the comprehensive robustness index with a preset comprehensive threshold value;

[0031] If the comprehensive robustness index is lower than the preset comprehensive threshold value, the output accuracy rate change rate S1, the F1 value fluctuation S2 and the recall rate deviation S3 in the performance index change are compared with the corresponding preset threshold values respectively; if any performance index change is greater than the corresponding preset threshold value, it is determined that the overall robustness of the recognition model does not meet the standard, and a defect label set is generated; the defect label set contains at least one of the source defect label, the propagation defect label and the content defect label;

[0032] If the comprehensive robustness index is not lower than the preset comprehensive threshold value, it is determined that the overall robustness of the recognition model meets the standard;

[0033] Wherein, the preset comprehensive threshold value and the corresponding preset threshold values of the output accuracy rate change rate S1, the F1 value fluctuation S2 and the recall rate deviation S3 are determined by statistical analysis of historical evaluation data.

[0034] As a preferred scheme of the present application, the dynamic adjustment of the recognition model parameters based on the defect label set through reinforcement learning comprises:

[0035] Based on the defect label set, the reinforcement learning agent selects an adjustment action for the recognition model parameters;

[0036] The adjustment action is executed to update the recognition model parameters;

[0037] Based on the updated recognition model, the robustness quantitative analysis is re-executed.

[0038] As a preferred scheme of the present application, the type of the adjustment action is determined by the label type in the defect label set, and at least one of the following is included:

[0039] If the defect label set contains the source defect label, the number of convolution kernels of the source feature extraction layer is increased;

[0040] If the defect label set contains the propagation defect label, a graph attention mechanism is added to the propagation path analysis module;

[0041] If the defect label set contains content defect labels, the text encoder is subjected to adversarial training, and the weight matrix of the word embedding layer, the query, key, and value projection matrices in the self-attention mechanism, and the learnable scaling factor and offset of the normalization layer are updated through gradient backpropagation.

[0042] As a preferred scheme of the present application, the adjustment action selected by the reinforcement learning agent for the identification model parameters comprises:

[0043] If the defect label set contains a single type of defect label, an adjustment action corresponding to the defect label type is selected.

[0044] If the defect label set contains multiple defect label types, a corresponding adjustment action is selected based on the preset priority of the defect label types and / or the correlation between the defect label types, and the correlation between the defect label types is obtained by statistical analysis of the historical intelligence message set.

[0045] A false threat intelligence identification model robustness quantitative analysis device comprises:

[0046] A data collection module is configured to extract source identifiers, propagation path node sequences, and message content texts from intelligence messages to construct a basic data set for robustness quantitative analysis.

[0047] A robustness test module is configured to modify the source identifiers, propagation path node sequences, and message content texts in the basic data set respectively and calculate the performance index changes of the identification model.

[0048] An index fusion module is configured to calculate a comprehensive robustness index based on the historical intelligence message set and the performance index changes through a weighted fusion algorithm.

[0049] A defect diagnosis module is configured to generate a defect label set based on the performance index changes and the comprehensive robustness index.

[0050] A model optimization module is configured to dynamically adjust the identification model parameters based on the defect label set through reinforcement learning.

[0051] Compared with the prior art, the beneficial effects of the present application are: through the three performance indicators of source modification, propagation path modification and content modification, the limitations of the prior art relying on a single performance indicator for identification model robustness quantitative analysis are overcome, and the identification model output accuracy is improved. Based on the dynamic calculation of the weight coefficient of the historical intelligence set, the comprehensive robustness index can respond to the latest threat trend in real time; the comprehensive robustness index, the output accuracy rate change rate S1, the F1 value fluctuation S2 and the recall rate deviation S3 are compared with the corresponding preset threshold respectively, a defect label set is generated to locate the weak module in the identification model, and the identification model is guided to adjust the parameters according to the defect label to improve the overall anti-performance of the identification model, thereby enhancing the anti-interference ability of the identification model in the complex network environment.

[0052] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0053] One or more embodiments are exemplified by the corresponding drawings, which do not constitute a limitation on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute a proportional limit, and wherein:

[0054] Figure 1 A flowchart of a false threat intelligence identification model robustness quantitative analysis method proposed for the embodiments of the present application;

[0055] Figure 2 A flowchart of a comprehensive robustness index calculation method proposed for the embodiments of the present application;

[0056] Figure 3 A modular structure schematic diagram of a false threat intelligence identification model robustness quantitative analysis device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0057] In order to be able to understand the features and technical content of the embodiments of the present disclosure more fully, the implementation of the embodiments of the present disclosure will be described in detail below, and the attached drawings are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, through multiple details, a sufficient understanding of the disclosed embodiments is provided. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to show.

[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0059] Unless otherwise stated, the term "multiple" means two or more.

[0060] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0061] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0062] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0063] Existing robustness analysis methods for identification models rely on a single performance metric, making it difficult to pinpoint specific weaknesses and lacking targeted parameter optimization mechanisms, thus hindering the model's adaptability to complex network environments. To address these issues, this invention proposes a robustness quantification analysis method for false threat intelligence identification models. This method overcomes the limitations of traditional methods that rely on a single performance metric by employing three performance metrics: source modification, propagation path modification, and content modification, thereby improving the accuracy of the identification model's output. Simultaneously, it dynamically calculates weight coefficients based on historical intelligence sets, making the comprehensive robustness metric more adaptable to dynamically evolving complex network environments. Based on the comparison results of the comprehensive robustness metric, the rate of change in output accuracy (S1), the F1 score fluctuation (S2), and the recall deviation (S3) with corresponding preset thresholds, a set of defect labels is generated to locate weak modules in the identification model and guide parameter adjustments, improving the model's performance in adversarial environments and enhancing its anti-interference capability in complex network environments.

[0064] like Figure 1 As shown, this is an embodiment of the present invention, which provides a method for robust quantitative analysis of a false threat intelligence identification model, including:

[0065] S1. Extract source identifiers, propagation path node sequences, and message content text from intelligence messages to construct the basic dataset for robust quantitative analysis; source identifiers include the identity recognition, authentication information, and credibility level of the intelligence issuing entity;

[0066] Further, the intelligence message can be obtained by calling an API interface of the threat intelligence sharing platform or parsing a log file, but is not limited to this way.

[0067] The source identifier can be obtained by parsing a metadata field of a header of the intelligence message, but is not limited to this way.

[0068] The propagation path node sequence reflects a propagation trajectory of the intelligence message in the network, and records an identifier of a node in the propagation trajectory and an interaction timestamp of the node. The propagation path node sequence can be extracted by analyzing a forwarding log or a timestamp sequence of the intelligence message. For example, an intelligence message can be propagated from a certain intelligence community to a threat intelligence system in an enterprise, and then forwarded to other cooperative platforms. Each propagation node can be a server, a database or a user terminal.

[0069] The message content text is a core part of the intelligence message, and is usually natural language text or structured data describing attack behaviors. When the message content text is extracted, the original message needs to be processed by word segmentation and noise removal, and core information related to threat intelligence, such as attack type and target address, is retained. The noise removal process can include removing irrelevant punctuation marks or formatting tags.

[0070] The extracted source identifier, propagation path node sequence and message content text are integrated into a structured data table to build a basic data set for robust quantitative analysis. For example, each row of the basic data set corresponds to an intelligence message, and includes three columns, namely the source identifier, the propagation path node sequence and the message content text.

[0071] In addition, to ensure data quality, the basic data set built can be preprocessed, such as data cleaning and standardization processing, wherein the data cleaning includes removing duplicate records or filling missing values.

[0072] S2, modifying data in the basic data set and calculating a performance index change of the identification model;

[0073] The data in the basic data set is modified, and a performance index change of the identification model is calculated, specifically including:

[0074] The source identifier in the basic data set is modified, and a change rate of output accuracy of the identification model is calculated S1. The modification of the source identifier in the basic data set includes replacing the source identifier, or deleting the source identifier, or forging the source identifier;

[0075] The nodes in the propagation path node sequence in the basic data set are modified, and a F1 value fluctuation S2 of the identification model is calculated.

[0076] Text adversarial samples are injected into the message content text in the basic data set, and a precision and recall deviation S3 of the identification model is calculated.

[0077] modifying a node in the sequence of nodes of the propagation path in the base dataset comprises:

[0078] selecting a node for deletion using random forest importance sampling; or, adding a pseudo node and forging an interaction timestamp.

[0079] injecting text adversarial samples into the message content text in the base dataset comprises:

[0080] replacing a synonym in the message content text by a pre-trained language model; or,

[0081] adjusting the syntax of the message content text; or,

[0082] inserting an interference sentence into the message content text.

[0083] Further, the recognition model is usually a deep learning-based classification model, but is not limited to this model, and is used to identify whether the intelligence message is a false threat intelligence.

[0084] The modification of the source identifier needs to simulate real scenarios, and can refer to common forgery patterns in historical data. In some embodiments, the forged identifier is generated according to the distribution of historical identity characteristics, for example, the forged enterprise domain name conforms to the MX record verification rule, so as to test the anti-forgery ability of the recognition model. At the same time, the modification ratio can be adjusted according to actual needs. For example, the original dataset contains 1000 intelligence messages, and the recognition model outputs an accuracy rate of 95%; after modifying the source identifier of 200 intelligence messages, the recognition model outputs an accuracy rate of 85%, and thus the change rate of the recognition model output accuracy rate The change rate quantifies the sensitivity of the recognition model to the modification of the source identifier.

[0085] The modification of the propagation path node can be implemented by deleting the node to generate a new propagation path node sequence. First, the importance of each node in the propagation path node sequence is evaluated by the random forest model, and the importance score of each node is calculated, wherein the node with a higher importance score is usually a key forwarding node. Subsequently, low-score nodes are removed based on the importance score to simulate the loss of non-critical nodes in the actual scenario. Finally, a new propagation path node sequence is generated and input into the identification model. For example, the original data set contains 1000 intelligence messages, and the propagation path of each intelligence message contains an average of 5 propagation nodes. After randomly deleting 1 propagation node, the identification model is re-run. Assuming that the original F1 value is 0.90 and the F1 value decreases to 0.85 after deleting the node, the F1 value fluctuation S2 is 0.05, which reveals the degree of dependence of the identification model on the integrity of the propagation path. In addition, the modification of the propagation path node can also be implemented by adding a pseudo node with a fake timestamp to construct an adversarial propagation path. For example, in the original path "node4→node5→node6→node7", a fake node fake_node is inserted to generate "node4→node5→node6→fake_node→node7", to simulate the interference scenario of the attacker implanting a false forwarding node.

[0086] The injection of text adversarial samples is used to simulate the scenario in which an attacker interferes with the identification model by modifying the content text in the intelligence message. For example, the original message content is "a certain IP address initiates DDoS attack", which may become "a certain IP address launches denial of service attack" after replacing synonyms; it may become "DDoS attack is initiated by a certain IP address" after adjusting the order; it may become "a certain IP address initiates DDoS attack, this attack can be ignored" after inserting a disturbance sentence. The recall bias is calculated by comparing the recall before and after modification. For example, the original recall is 0.98, which decreases to 0.88 after injecting adversarial samples, and the bias S3 is 0.10. In addition, the injection ratio and the generation method of the adversarial samples can be adjusted according to the test requirements.

[0087] By modifying the basic data set, the adversarial interference that may occur in the real scenario is simulated, and then the performance of the identification model under different interference is evaluated. The modification operation is carried out in three aspects of source identifier, propagation path node sequence and message content text, to comprehensively detect the robustness of the model.

[0088] S3, based on the historical intelligence message set and the performance index change, the comprehensive robustness index is calculated by a weighted fusion algorithm;

[0089] In some embodiments, as shown in Figure 2 based on the historical intelligence message set and the performance index change, the comprehensive robustness index is calculated by a weighted fusion algorithm, including:

[0090] S31, Obtain a set of historical intelligence messages, including false threat intelligence samples, real threat intelligence samples, and corresponding robustness evaluation results of the identification model within a preset time period;

[0091] S32, based on the historical intelligence message set, uses a weighted linear regression model to calculate the weight coefficients, including source modification weight W1, propagation path modification weight W2 and content modification weight W3;

[0092] S33, based on the weighting coefficients, calculates the comprehensive robustness index T using a weighted fusion algorithm. The calculation formula is as follows:

[0093] T = W1 × S1 + W2 × S2 + W3 × S3

[0094] Where W1+W2+W3=1.

[0095] In some embodiments, the weighting coefficients are determined through the following calculation steps, but are not limited to this method. Specific calculation steps include:

[0096] In the historical intelligence message set, the i-th type of modification operation is executed K times;

[0097] Record the changes in performance metrics after each modification.

[0098] Calculate the average impact of the i-th type of modification.

[0099] Calculate the weighting coefficients

[0100] Among them, P m This represents the average impact value of the m-th type of modification.

[0101] Assuming the i=1th modification operation is to forge the source identifier, the i=2nd modification operation is to modify the propagation node, and the i=3rd modification operation is to inject adversarial text, and each type of modification operation is executed independently 4 times, the performance metrics change

[0102] Calculate the average impact value of type 1 modifications.

[0103] Calculate the average impact value of type 2 modifications.

[0104] Calculate the average impact value of type 3 modifications.

[0105] Calculate the weighting coefficients

[0106] Calculate the weighting coefficients

[0107] Calculate the weight coefficient

[0108] Further, the weight coefficient R i The defense weight of the current maximum threat modification type is adjusted in real time with the update of the historical intelligence message set. The historical intelligence message set needs to cover typical modification scenarios, such as different forgery rules and counter-text types.

[0109] The weight coefficient reflects the contribution of different performance indicators to the comprehensive robustness, and the level of the comprehensive robustness indicator reflects the overall anti-interference ability of the identification model, which can respond to the latest threat trends in real time.

[0110] S4, generating a defect label set based on the performance indicator changes and the comprehensive robustness indicator;

[0111] The comprehensive robustness indicator is compared with a preset comprehensive threshold;

[0112] If the comprehensive robustness indicator is lower than the preset comprehensive threshold, the output accuracy rate change rate S1, the F1 value fluctuation S2, and the recall rate deviation S3 in the performance indicator changes are compared with the corresponding preset thresholds respectively; if any performance indicator change is greater than the corresponding preset threshold, it is determined that the overall robustness of the identification model is not up to standard, and a defect label set is generated; the defect label set includes at least one of the source defect label, the propagation defect label, and the content defect label;

[0113] If the comprehensive robustness indicator is not lower than the preset comprehensive threshold, it is determined that the overall robustness of the identification model is up to standard.

[0114] The preset comprehensive threshold and the preset thresholds corresponding to the output accuracy rate change rate S1, the F1 value fluctuation S2, and the recall rate deviation S3 are determined by statistical analysis of historical evaluation data.

[0115] Further, if T < T0, the output accuracy rate change rate S1, the F1 value fluctuation S2, and the recall rate deviation S3 are compared with the corresponding preset thresholds respectively If Or, Or, The overall robustness of the identification model is determined to be not up to standard, and a defect label set is generated.

[0116] Wherein, T is the comprehensive robustness indicator, T0 is the preset comprehensive threshold, is the preset threshold of the output accuracy rate change rate, is the preset threshold of the F1 value fluctuation, is the preset threshold of the recall rate deviation.

[0117] For example, if T < T0, and It is determined that the overall robustness of the identification model does not meet the standard, and a defect label set is generated, which only contains the source defect label.

[0118] If T < T0, and It is determined that the overall robustness of the identification model does not meet the standard, and a defect label set is generated, which contains the source defect label and the propagation defect label.

[0119] The defect label set includes a single type of defect label and multiple types of defect labels, and the representation form can be a list or a vector, such as {source defect, content defect}.

[0120] Through the determination of the defect label, the weak module in the identification model can be located, and specific targets are provided for the subsequent adjustment of the identification model parameters, thereby avoiding blind optimization.

[0121] S5, based on the defect label set, dynamically adjusting the identification model parameters through reinforcement learning.

[0122] Based on the defect label set, the identification model parameters are dynamically adjusted through reinforcement learning, specifically including:

[0123] Based on the defect label set, the reinforcement learning agent selects an adjustment action for the identification model parameters;

[0124] The selected adjustment action is executed to update the identification model parameters;

[0125] Based on the updated identification model, the robustness quantification analysis is re-executed.

[0126] The type of adjustment action is determined by the label type in the defect label set, including at least one of the following:

[0127] If the defect label set contains the source defect label, the number of convolution kernels of the source feature extraction layer is increased;

[0128] If the defect label set contains the propagation defect label, a graph attention mechanism is added to the propagation path analysis module;

[0129] If the defect label set contains the content defect label, the text encoder is subjected to adversarial training, and the weight matrix of the word embedding layer, the query, key, and value projection matrices in the self-attention mechanism, and the learnable scaling factor and offset of the normalization layer are updated through gradient backpropagation.

[0130] The reinforcement learning agent selects an adjustment action for the identification model parameters, including:

[0131] If the defect label set contains a single type of defect label, an adjustment action corresponding to the defect label type is selected.

[0132] If the defect label set contains multiple defect label types, the corresponding adjustment action is selected based on the priority of the preset defect label type and / or the correlation between the defect label types, which is obtained by statistical analysis of the historical intelligence message set.

[0133] Further, the source feature extraction layer is usually a convolutional neural network for extracting key features from the source identifier, such as the identity information and credibility score of the publisher. Increasing the number of convolution kernels can enhance the recognition model's ability to capture source features. For example, after increasing the original number of convolution kernels from 32 to 64, the recognition model can extract richer feature patterns, thereby better distinguishing between real and fake source identifiers, improving the recognition model's anti-interference ability to source modification, and reducing the misjudgment rate.

[0134] The propagation path analysis module is usually based on a graph neural network to model the propagation path of the intelligence message. Adding a graph attention mechanism involves introducing an attention layer in the propagation path analysis module to assign different attention weights to each node, enhancing the recognition model's focus on key nodes and improving the recognition model's robustness to propagation path modification, reducing misjudgment caused by loss of propagation path information.

[0135] The text encoder is usually based on a pre-trained language model such as BERT to extract semantic features of the message content text. Adversarial training involves introducing adversarial samples during training and updating the weight matrix of the word embedding layer, the query, key, and value projection matrices in the self-attention mechanism, and the learnable scaling factor and offset of the normalization layer through gradient backpropagation, enhancing the recognition model's adaptability to text content modification.

[0136] When the defect label set contains multiple defect types, the priority of the preset defect label type can be fixed, for example: source defect > propagation defect > content defect, or dynamically generated according to actual conditions.

[0137] In some embodiments, for dynamically generating the priority of the preset defect label type, the specific implementation steps include:

[0138] Calculate the severity of the ith defect:

[0139] wherein, is the preset threshold value corresponding to the ith defect; N i is the weight of the ith defect, obtained by statistical analysis of historical data;

[0140] Based on the calculated severity Q i , from high to low, to determine the priority of the preset defect label type.

[0141] The correlation between the defect label types is obtained through statistical analysis of a historical intelligence message set, and is specifically defined as follows:

[0142] When the performance index of the defect label type B does not decrease but increases after the adjustment action corresponding to the defect label type A is performed, and the increase is greater than a preset increase threshold, the correlation is positive correlation; the preset increase threshold needs to be reasonably set according to a specific application scenario.

[0143] When the performance index of the defect label type B deteriorates after the adjustment action corresponding to the defect label type A is performed, and the deterioration is greater than a preset deterioration threshold, the correlation is negative correlation; the preset deterioration threshold needs to be reasonably set according to a specific application scenario.

[0144] If the correlation between the two defects is positive correlation, then the corresponding adjustment action is selected in turn according to the priority of the preset defect label type; if the correlation between the two defects is negative correlation, then the conflict needs to be solved first, and then the corresponding adjustment action is selected in turn according to the priority of the preset defect label type.

[0145] For example, since the correlation between the source defect and the propagation defect is positive correlation, if the defect label set contains the source defect label and the propagation defect label, then the corresponding adjustment action is selected in turn according to the priority of the preset defect label type, that is, the number of convolution kernels is first increased, and then the graph attention mechanism is added to the propagation path analysis module.

[0146] Since the correlation between the source defect and the content defect is negative correlation, if the defect label set contains the source defect label and the content defect label, then the gradient conflict needs to be solved by using the gradient projection method first, and then the corresponding adjustment action is selected in turn according to the priority of the preset defect label type. The specific steps of solving the gradient conflict by using the gradient projection method are as follows:

[0147] The cosine similarity of the source optimization gradient and the content optimization gradient is calculated;

[0148] If the cosine similarity is less than 0, then the content optimization gradient is projected to the orthogonal complement space of the source gradient;

[0149] Since the correlation between the propagation defect and the content defect is negative correlation, if the defect label set contains the propagation defect label and the content defect label, then the structure conflict needs to be solved by inserting a dimension adaptation layer first, and then the corresponding adjustment action is selected in turn according to the priority of the preset defect label type. The specific steps of solving the structure conflict by inserting the dimension adaptation layer are as follows:

[0150] A fully connected adaptation layer is inserted at the output end of the propagation path analysis module to which the graph attention mechanism has been added;

[0151] Freezing the propagation path analysis module parameters outside the fully connected adaptive layer.

[0152] Performing the selected adjustment action to update the parameters of the identification model; based on the updated identification model, re-executing the robustness quantification analysis. The process of re-executing the robustness quantification analysis is consistent with the process of the initial robustness quantification analysis, including obtaining intelligence messages and constructing a basic data set for robustness quantification analysis, modifying the basic data set, calculating performance index changes and comprehensive robustness indexes.

[0153] According to the adjustment of the identification model parameters guided by the defect label set, the anti-interference ability of the identification model in the complex network environment is improved. Re-executing the robustness quantification analysis on the updated identification model can find new defect labels, which provides a basis for the next round of optimization.

[0154] As shown in Figure 3 A false threat intelligence identification model robustness quantification analysis device, comprising:

[0155] A data collection module for extracting source identifiers, propagation path node sequences and message content texts from intelligence messages to construct a basic data set for robustness quantification analysis;

[0156] A robustness test module for modifying the source identifiers, propagation path node sequences and message content texts in the basic data set respectively, and calculating the performance index changes of the identification model;

[0157] An index fusion module for calculating a comprehensive robustness index based on the historical intelligence message set and the performance index changes through a weighted fusion algorithm;

[0158] A defect diagnosis module for generating a defect label set based on the performance index changes and the comprehensive robustness index;

[0159] A model optimization module for dynamically adjusting the parameters of the identification model based on the defect label set through reinforcement learning.

[0160] To sum up, the application quantitatively analyzes the robustness of the identification model through multiple performance indicators such as source modification, propagation path modification and content modification, overcomes the limitation that the existing method only relies on a single performance indicator for robustness evaluation, and improves the output accuracy of the identification model. Through dynamic calculation of the weight coefficient, the comprehensive robustness index can respond to the latest threat trend in real time. In addition, the comprehensive robustness index, the output accuracy rate change rate S1, the F1 value fluctuation S2 and the recall rate deviation S3 are compared with the respective preset threshold, and the corresponding defect label set is generated according to the comparison result, so as to accurately locate the weak module in the identification model. According to the defect label, the identification model is guided to adjust the parameters, effectively improves the anti-performance, and further enhances the anti-interference ability of the identification model in the complex network environment.

[0161] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, but that the application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the application being defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalents of the essential elements of the claims are intended to be embraced therein.

Claims

1. A method for robust quantitative analysis of a false threat intelligence identification model, characterized in that, include: Source identifiers, propagation path node sequences, and message content text are extracted from intelligence messages to construct a basic dataset for robust quantitative analysis; the source identifiers include the identity recognition, authentication information, and credibility level of the intelligence issuing entity; Modify the data in the base dataset and calculate the changes in the performance metrics of the recognition model; the changes in the performance metrics include the rate of change of output accuracy S1, the fluctuation of F1 value S2, and the recall deviation S3; Based on historical intelligence message sets and performance index changes, a comprehensive robustness index is calculated using a weighted fusion algorithm; A set of defect labels is generated based on changes in performance indicators and comprehensive robustness indicators; The defect tag set generated based on performance index changes and comprehensive robustness index includes: The comprehensive robustness index is compared with the preset comprehensive threshold. If the overall robustness index is lower than the preset overall threshold, the output accuracy change rate S1, F1 value fluctuation S2, and recall deviation S3 among the performance index changes are compared with the corresponding preset thresholds respectively; if the change of any performance index is greater than the corresponding preset threshold, it is determined that the overall robustness of the recognition model has not met the standard, and a defect tag set is generated; the defect tag set includes at least one of source defect tag, propagation defect tag, and content defect tag. If the overall robustness index is not lower than the preset overall threshold, the overall robustness of the recognition model is deemed to meet the standard. Among them, the preset comprehensive threshold and the preset thresholds corresponding to the output accuracy change rate S1, F1 value fluctuation S2, and recall deviation S3 are determined by statistical analysis of historical evaluation data. Based on the defect label set, the parameters of the recognition model are dynamically adjusted through reinforcement learning; The dynamic adjustment of the identification model parameters based on the defect label set through reinforcement learning includes: Based on the defect label set, the reinforcement learning agent selects adjustment actions for the recognition model parameters; the type of the adjustment action is determined by the label type in the defect label set. Execute the selected adjustment action to update the recognition model parameters; Robust quantization analysis was re-performed based on the updated recognition model.

2. The robustness quantification analysis method for a false threat intelligence identification model according to claim 1, characterized in that, The modification of data in the base dataset and the calculation of changes in the performance metrics of the recognition model include: Modify the source identifier in the base dataset, including replacing the source identifier, deleting the source identifier, or forging the source identifier, and calculate the rate of change S1 of the output accuracy of the recognition model; Modify the nodes in the propagation path node sequence in the basic dataset, and calculate the F1 value fluctuation S2 of the recognition model; Inject text adversarial samples into the message content text in the basic dataset and calculate the recall bias S3 of the recognition model.

3. The robustness quantification analysis method for a false threat intelligence identification model according to claim 2, characterized in that, The nodes in the propagation path node sequence in the modified basic dataset include: Nodes can be selected for deletion using random forest importance sampling; alternatively, pseudo-nodes can be added and fake interaction timestamps can be created.

4. The robustness quantification analysis method for a false threat intelligence identification model according to claim 3, characterized in that, The injection of text adversarial samples into the message content text in the basic dataset includes: Replace synonyms in the message content text using a pre-trained language model; or... Adjust the word order of the message content text; or, Insert distractor sentences into the message content text.

5. The robustness quantification analysis method for a false threat intelligence identification model according to claim 4, characterized in that, The comprehensive robustness index calculated using a weighted fusion algorithm based on historical intelligence message sets and performance index changes includes: Acquire historical intelligence message sets, including false threat intelligence samples, real threat intelligence samples, and corresponding robustness evaluation results of the identification model within a preset time period; Based on historical intelligence message sets, a weighted linear regression model is used to calculate weight coefficients, including source modification weight W1, propagation path modification weight W2, and content modification weight W3. Based on the weighting coefficients, the comprehensive robustness index T is calculated using a weighted fusion algorithm. The calculation formula is as follows: in, .

6. The robustness quantification analysis method for a false threat intelligence identification model according to claim 5, characterized in that, The type of adjustment action is determined by the label type in the defect label set, including at least one of the following: If the defect label set contains source defect labels, then increase the number of convolutional kernels in the source feature extraction layer; If the defect label set contains propagating defect labels, then add a graph attention mechanism to the propagation path analysis module; If the defect label set contains content defect labels, then the text encoder is subjected to adversarial training, and the weight matrix of the word embedding layer, the query, key, and value projection matrices in the self-attention mechanism, and the learnable scaling factor and offset of the normalization layer are updated through gradient backpropagation.

7. The robustness quantification analysis method for a false threat intelligence identification model according to claim 6, characterized in that, The reinforcement learning agent selection includes the following adjustments to the recognition model parameters: If the defect label set contains a single type of defect label, then select the adjustment action corresponding to that defect label type; If the defect tag set contains multiple defect tag types, then based on the preset priority of the defect tag types and / or the association between the defect tag types, the corresponding adjustment action is selected; the association between the defect tag types is obtained through statistics from historical intelligence message sets.

8. A robustness quantification analysis device for a false threat intelligence identification model, comprising a robustness quantification analysis method for a false threat intelligence identification model according to any one of claims 1 to 7, characterized in that, include: The data collection module is used to extract source identifiers, propagation path node sequences, and message content text from intelligence messages to build a basic dataset for robust quantitative analysis. The robustness testing module is used to modify the source identifier, propagation path node sequence, and message content text in the basic dataset, and calculate the changes in the performance metrics of the recognition model. The indicator fusion module is used to calculate a comprehensive robustness indicator based on historical intelligence message sets and performance indicator changes through a weighted fusion algorithm. The defect diagnosis module is used to generate a set of defect tags based on changes in performance indicators and comprehensive robustness indicators; The model optimization module is used to dynamically adjust the parameters of the recognition model based on the defect label set through reinforcement learning.

Images