Threat intelligence question answering method, device and equipment based on attention bias enhancement

Abstract

The application provides a threat intelligence question and answer method, device and equipment based on attention bias enhancement, and relates to the technical field of artificial intelligence and network security. The method comprises the following steps: obtaining input threat intelligence query information, and generating query semantic representation vector; based on the query semantic representation vector, through hybrid retrieval of semantic matching and keyword matching, a candidate intelligence document set is obtained from an external threat intelligence library; the candidate intelligence document set is input into a supervised sorting model which has been trained, and the relevance scores of each candidate document are output, and a selected document set is screened based on the relevance scores; an improved large language model is obtained based on the relevance scores corresponding to the selected documents; each selected document is input into the improved large language model as context, and a zero-shot chain thinking prompt strategy is adopted to guide the improved large language model to generate answers. The application can accurately focus on key evidence in threat intelligence question and answer, and improves the accuracy and reliability of the question and answer results.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cybersecurity technology, and in particular to a threat intelligence question answering method, apparatus and device based on attention bias enhancement. Background Technology

[0002] The current cybersecurity threat landscape is becoming increasingly complex. Cybersecurity threat defense systems rely on cybersecurity threat intelligence for threat identification and attribution analysis to promptly identify potential risks and formulate defense strategies. Large language models are increasingly being introduced into security applications such as threat intelligence analysis, attack posture assessment, and automated question answering, providing new technical support for intelligent system defense. However, large threat intelligence models, fine-tuned from historical cybersecurity threat data, have knowledge boundaries limited to a specific point in time. They cannot dynamically synchronize with advanced cybersecurity threat intelligence, resulting in poor real-time performance and difficulty in addressing rapidly evolving threat scenarios such as new Advanced Persistent Threat (APT) attacks, zero-day vulnerabilities, and variant malware samples.

[0003] In existing threat intelligence question-answering systems, Retrieval-Augmented Generation (RAG) technology is widely used to compensate for the lag in knowledge updates of large models. However, traditional vector retrieval mechanisms still have significant limitations in threat intelligence scenarios. Due to the complex structures and fragmented fields of security texts such as attack chains, Indicators of Compromise (IOC), and Common Vulnerabilities and Exposures (CVE), relying solely on semantic vector similarity often fails to accurately capture query intent, easily leading to false positives of "high similarity, low relevance." Furthermore, intelligence texts are typically lengthy. With limited context windows in large models, directly inputting large amounts of retrieval content into the model in a coarse-grained manner wastes window resources and interferes with model attention allocation, resulting in the loss of crucial evidence and affecting the accuracy of inference results.

[0004] Furthermore, when faced with unknown threat samples, large language models, due to their end-to-end output mechanism and lack of auditable reasoning processes, cannot clearly explain the formation logic of threat events. The models may generate illusory content to mislead security analysts into making incorrect judgments. Summary of the Invention

[0005] This invention provides a threat intelligence question answering method, apparatus, and device based on attention bias enhancement to solve the problems of inaccurate screening, low credibility of answers, and inability to trace the source.

[0006] In a first aspect, embodiments of the present invention provide a threat intelligence question-answering method based on attention bias enhancement, including: Obtain the input threat intelligence query information and generate a query semantic representation vector; Based on the query semantic representation vector, a candidate intelligence document set is obtained from an external threat intelligence database through a hybrid retrieval method that combines semantic matching and keyword matching. The candidate intelligence document set is input into the trained supervised ranking model, which outputs the relevance score of each candidate document, and a selected document set is selected based on the relevance score. The relevance scores of the selected documents in the selected document set are converted into word-level attention bias matrices, which are then superimposed on the original attention scores of the large language model to obtain an improved large language model. Each selected document is used as context input to the improved large language model, and a zero-shot chain-like thinking prompting strategy is adopted to guide the improved large language model to generate answers.

[0007] In one possible implementation, based on the query semantic representation vector, a candidate intelligence document set is obtained from an external threat intelligence database through a hybrid retrieval method combining semantic matching and keyword matching, including: Semantic encoding is performed on each intelligence document in the external threat intelligence database to generate a semantic representation vector for each intelligence document; Calculate the similarity between the query semantic representation vector and the semantic representation vector of each intelligence document to obtain the semantic matching score of each intelligence document; Based on the keywords in the threat intelligence query information, the keyword matching score of each intelligence document is calculated using the term matching method. For any intelligence document, the keyword matching score and the semantic matching score of the intelligence document are weighted and fused to obtain the comprehensive retrieval score of the intelligence document. Based on the comprehensive retrieval score of each intelligence document, the intelligence documents are sorted in descending order, and the first first preset number of intelligence documents are selected to form the candidate intelligence document set.

[0008] In one possible implementation, before inputting the candidate intelligence document set into the trained supervised ranking model, the method further includes: Obtain a training dataset, wherein the training samples in the training dataset include labeled query information, positive sample documents, negative sample documents, and semantically related but actually irrelevant sample documents; Construct a cross-coding sorting model; Using the training dataset, the cross-coding ranking model is trained to obtain the trained supervised ranking model; The loss function includes a sorting cross-entropy loss term and an interval loss term. The sorting cross-entropy loss term is used to increase the sorting probability of the positive sample documents, and the interval loss term is used to force the relevance score of the positive sample documents to be higher than the relevance score of the semantically related but actually unrelated sample documents by an interval threshold.

[0009] In one possible implementation, the loss function is calculated as follows: ; in, Let the loss function be... The sorting cross-entropy loss term is... For the interval loss term, The relevance score of the positive sample document. For the first The relevance score of each of the negative sample documents. For the first The relevance score of the semantically related but actually irrelevant sample documents. To lose weight, The interval threshold is... It is an exponential function. It is a function for maximizing the value.

[0010] In one possible implementation, the candidate intelligence document set is input into a trained supervised ranking model, and the relevance scores of each candidate document are output to rank the candidate documents, thereby selecting a curated document set, including: For any candidate document in the candidate intelligence document set, the query semantic representation vector is concatenated with the candidate document to construct the input sequence corresponding to the candidate document; Each input sequence is input into the trained supervised ranking model, which outputs a relevance score for each candidate document. The candidate documents are sorted in descending order based on their relevance scores. The first second preset number of candidate documents are selected to form the selected document set.

[0011] In one possible implementation, the relevance scores corresponding to the selected documents in the selected document set are converted into a word-level attention bias matrix, which is then superimposed on the original attention score of the large language model to obtain an improved large language model, including: For any selected document in the selected document set, the relevance score of the selected document is standardized and mapped to the document-level attention bias coefficient of the selected document. Each document-level attention bias coefficient is extended to each word of the corresponding selected document to obtain a word-level attention bias matrix; In each attention layer of the large language model, the word-level attention bias matrix is ​​superimposed on the original attention score to obtain the biased attention score. The output of each attention layer is calculated based on the biased attention score to obtain the improved large language model.

[0012] In one possible implementation, the formula for calculating the word-level attention bias matrix is: ; ; in, The word-level attention bias matrix The Middle Line 1 q Column elements, as a word element q The document-level bias coefficient corresponding to the document. For the first The document-level attention bias coefficient of each of the selected documents. The bias strength coefficient, For the first The relevance score of the selected documents The mean of all the relevance scores in the selected document set. The standard deviation of all the relevance scores in the selected document set.

[0013] In one possible implementation, the formula for calculating the biased attention score is: ; in, The biased attention score. For normalization function, For querying the matrix, The key matrix, For value matrices, The feature dimension corresponding to the attention head. is the word-level attention bias matrix.

[0014] Secondly, embodiments of the present invention provide a threat intelligence question-answering device based on attention bias enhancement, comprising: The input parsing module is used to obtain the input threat intelligence query information and generate a query semantic representation vector; The candidate intelligence document filtering module is used to obtain a set of candidate intelligence documents from an external threat intelligence database by performing a hybrid retrieval of semantic matching and keyword matching based on the query semantic representation vector. The selected document filtering module is used to input the candidate intelligence document set into the trained supervised ranking model, output the relevance score of each candidate document, and filter out the selected document set based on the relevance score. The large language model improvement module is used to convert the relevance scores of the selected documents in the selected document set into a word-level attention bias matrix, which is then superimposed on the original attention score of the large language model to obtain an improved large language model. The answer generation module is used to input various selected documents as context into the improved large language model, and adopts a zero-shot chain-like thinking prompting strategy to guide the improved large language model to generate answers.

[0015] Thirdly, embodiments of the present invention provide a threat intelligence question-answering device based on attention bias enhancement, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the threat intelligence question-answering method based on attention bias enhancement as described in the first aspect or any possible implementation of the first aspect.

[0016] In this embodiment of the invention, a set of candidate intelligence documents is input into a trained supervised ranking model. The relevance scores of each candidate document are output to rank the candidate documents, and a selected set of documents is selected. The trained supervised ranking model removes noise of "high similarity and low relevance" to improve the accuracy of the selection. The relevance scores of the selected documents in the selected document set are converted into a word-level attention bias matrix and superimposed on the original attention score of the large language model to obtain an improved large language model. This allows the large language model to dynamically acquire the latest threat knowledge and automatically focus on key evidence during the reasoning process, improving the large language model's ability to identify complex threat events and the effectiveness of reasoning. By introducing a zero-shot chain-like thinking prompt strategy, the model can generate logically coherent and evidence-supported answers, which facilitates tracing the source and improves the transparency and credibility of reasoning. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of the threat intelligence question answering method based on attention bias enhancement provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the threat intelligence question answering device based on attention bias enhancement provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the threat intelligence question-answering device based on attention bias enhancement provided in an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] like Figure 1 The threat intelligence question answering method based on attention bias enhancement provided in this embodiment of the invention includes: Step 101: Obtain the input threat intelligence query information and generate a query semantic representation vector.

[0020] In this embodiment, step 101 specifically includes: The input threat intelligence query information is fed into a pre-trained language model to generate a semantic representation vector, or the threat intelligence query information is transformed into a semantic representation vector through other semantic embedding methods.

[0021] Step 102: Based on the query semantic representation vector, a set of candidate intelligence documents is obtained from the external threat intelligence database through a hybrid retrieval of semantic matching and keyword matching.

[0022] In one embodiment, step 102 includes: Semantic encoding is performed on each intelligence document in the external threat intelligence database to generate a semantic representation vector for each intelligence document.

[0023] The similarity between the query semantic representation vector and the semantic representation vector of each intelligence document is calculated to obtain the semantic matching score of each intelligence document.

[0024] Based on the keywords in the threat intelligence query information, the keyword matching score of each intelligence document is calculated using the term matching method.

[0025] For any intelligence document, the keyword matching score and the semantic matching score of the intelligence document are weighted and fused to obtain the comprehensive retrieval score of the intelligence document.

[0026] Based on the comprehensive retrieval score of each intelligence document, the intelligence documents are sorted in descending order, and the first preset number of intelligence documents are selected to form a candidate intelligence document set.

[0027] This step employs a weighted fusion of semantic retrieval and keyword retrieval in a dual-path search, balancing semantic generalization and precision matching in threat intelligence queries, thereby improving the quality of the candidate intelligence document set.

[0028] Step 103: Input the candidate intelligence document set into the trained supervised ranking model, output the relevance score of each candidate document, and select the best document set based on the relevance score.

[0029] In one embodiment, prior to step 103, the method further includes: Step 102a: Obtain the training dataset. The training samples in the training dataset include labeled query information, positive sample documents, negative sample documents, and semantically related but actually irrelevant sample documents.

[0030] Construct a cross-coding sorting model.

[0031] The cross-coding ranking model is trained using the training dataset to obtain a trained supervised ranking model.

[0032] The loss function includes a ranking cross-entropy loss term and a margin loss term. The ranking cross-entropy loss term is used to increase the ranking probability of positive sample documents, while the margin loss term is used to force the relevance score of positive sample documents to be higher than the relevance score of semantically related but actually unrelated sample documents by a margin threshold.

[0033] This step constructs a training dataset that includes semantically related but actually unrelated sample documents, enabling the supervised ranking model to acquire refined relevance discrimination capabilities in threat intelligence scenarios, fundamentally solving the false detection problem of high similarity and low relevance in traditional retrieval.

[0034] In this embodiment, the formula for calculating the loss function is: ; in, For loss function, For sorting cross-entropy loss term, For interval loss term, The relevance score for positive sample documents. For the first The relevance score of each negative sample document. For the first The relevance score of semantically related but actually irrelevant sample documents. To lose weight, This is the interval threshold. It is an exponential function. It is a function for maximizing the value.

[0035] The loss function, through joint optimization of the ranking cross-entropy loss term and the interval loss term, ensures the basic ranking ability of the supervised ranking model while forcibly widening the score gap between positive sample documents and semantically related but actually irrelevant sample documents. This enables the supervised ranking model to identify semantically related but actually irrelevant sample documents, fundamentally suppressing the false detection problem of high similarity and low relevance in threat intelligence scenarios.

[0036] In this embodiment, the training process of the supervised ranking model specifically includes: The BAAI / bge-reranker-base model is used as the cross-coding ranking model.

[0037] Obtain the training dataset, which includes labeled positive sample documents, negative sample documents, and semantically related but actually irrelevant sample documents (Hard Negative).

[0038] For any training document in the training dataset , query semantic representation vector Concatenate with the training document to construct the cross-encoding input: ; in, For the first One cross-encoded input, For classification labels, Used as a separator.

[0039] After inputting the cross-encoder into the Transformer encoder, a sentence-level representation is obtained: ; in, for Sentence-level representation vectors of cross-coded inputs (⋅) is the Transformer encoder function. These are the parameters for the Transformer encoder.

[0040] The relevance score of training documents is calculated by linear stratification: ; in, for The relevance score of each cross-encoded input. For the ranking layer weights, This is the bias for the sorting layer.

[0041] For the same query information, the relevance scores of multiple corresponding training documents are normalized to obtain the relevance probability distribution: ; in, For the first The correlation probability of each cross-encoded input. For the first The relevance score of each cross-encoded input. This represents the total number of training documents.

[0042] Based on the probability distribution, the ranking score is further linked to the attention bias of the subsequent large language model, so that the ranking model is not only used for document reordering, but also for guiding the evidence focus in the reasoning stage of the large model.

[0043] The training dataset is constructed based on real-world scenario templates for four data types: IP, domain, URL, and vulnerability. It includes three types of samples: Positive samples are real or high-confidence threat intelligence documents corresponding to the query; Negative samples are topic-irrelevant or low-relevance documents; and Hard Negative samples are documents that scored high in the initial coarse search phase but do not match the true intent of the query.

[0044] set up Scoring for Positive documents Score the Negative documentation. Score the Hard Negative documents and define the loss function based on the scores.

[0045] Simultaneously optimize Transformer encoder parameters through backpropagation. Ranking layer weights Bias with sorting layer This enables supervised ranking models to learn more refined relevance discrimination capabilities in threat intelligence scenarios.

[0046] In one embodiment, step 103 includes: For any candidate document in the candidate intelligence document set, the query semantic representation vector is concatenated with the candidate document to construct the input sequence corresponding to the candidate document.

[0047] Each input sequence is fed into the trained supervised ranking model, which outputs a relevance score for each candidate document.

[0048] Candidate documents are sorted in descending order based on their relevance score.

[0049] Select the first second preset number of candidate documents to form a curated document set.

[0050] This step calculates a relevance score for each candidate document using the trained supervised ranking model and sorts and filters them, accurately recalling highly relevant documents and effectively removing low-relevance documents, thus reducing the contextual input size of the large language model. At the same time, it provides an effective source for the attention bias calculation of the large language model.

[0051] Step 104: Convert the relevance scores of the selected documents in the selected document set into word-level attention bias matrices, and superimpose them onto the original attention scores of the large language model to obtain an improved large language model.

[0052] In one embodiment, step 104 includes: For any selected document in the selected document set, the relevance score of the selected document is standardized and mapped to the document-level attention bias coefficient of the selected document.

[0053] Each document-level attention bias coefficient is extended to each word in the corresponding selected document to obtain a word-level attention bias matrix.

[0054] In each attention layer of the large language model, the word-level attention bias matrix is ​​superimposed on the original attention score to obtain the biased attention score.

[0055] The output of each attention layer is calculated based on the biased attention score to obtain an improved large language model.

[0056] This step transforms the relevance scores of selected documents into a lexical-level attention bias matrix, which plays a role in each layer of attention calculation in the large language model. This enables the regulation of attention scores from the document level to the lexical level, allowing the large language model to focus on key evidence.

[0057] In this embodiment, the formula for calculating the word-level attention bias matrix is ​​as follows: ; ; in, Lexical level attention bias matrix The Middle Line 1 q Column elements, as a word element q The document-level bias coefficient corresponding to the document. For the first Document-level attention bias coefficients for selected documents. The bias strength coefficient, For the first The relevance score of each selected document. This is the mean of all relevance scores in the selected document collection. This represents the standard deviation of all relevance scores in the selected document collection.

[0058] The lexical-level attention bias matrix is ​​obtained by standardizing relevance scores and mapping from document level to lexical level, ensuring that each lexical in the selected documents receives reasonable attention enhancement.

[0059] In this embodiment, the formula for calculating the biased attention score is: ; in, The attention score after bias. For normalization function, For querying the matrix, The key matrix, For value matrices, The feature dimension corresponding to the attention head. This is the word-level attention bias matrix.

[0060] The formula for calculating the biased attention score directly superimposes the word-level attention bias matrix onto the original attention score, guiding the large language model to focus on key evidence words in highly relevant documents at each attention layer.

[0061] In this embodiment, step 104 specifically includes: The relevance score distribution of the selected document collection is as follows: ,in, This represents the total number of selected documents in the selected document collection.

[0062] The relevance score distribution of the selected document set is standardized to obtain the document-level attention bias coefficient. To further influence the reasoning process of the large language model by allowing the ranking of selected documents to influence the overall process, document-level attention bias coefficients are mapped to token-level attention bias matrices. This allows large language models to assign higher attention weights to key documents with higher scores and lower attention weights to documents with lower scores during inference, thereby improving the model's focus on key evidence within a limited context window. Specifically, the document bias is copied to all tokens within it to form a token-level bias vector. Then, an attention bias matrix is ​​constructed. .

[0063] Attention bias matrix It will be used as an additional attention score item, and will be included in the calculation of the inference stage along with the native attention logits of the large language model.

[0064] Step 105: Using each selected document as contextual input to the improved large language model, a zero-shot chain-like thinking prompting strategy is adopted to guide the improved large language model to generate answers.

[0065] In this embodiment, step 105 specifically includes: Based on the order of the selected documents in the selected document set, the query semantic representation vector is concatenated with each selected document in turn to form the context, and then input into the improved large language model.

[0066] During the generation phase, a zero-sample chain-thinking prompting strategy is adopted to guide the model to gradually generate reasoning processes and conclusions, thereby achieving logical consistency and enhanced interpretability.

[0067] For example, in high-risk or unreliable scenarios, a few-sample chain thinking strategy can be adopted, which introduces several high-quality examples in the prompts to enhance the stability of reasoning and the consistency of output.

[0068] The following specific embodiments illustrate the threat intelligence question answering method based on attention bias enhancement provided by the present invention: Based on the generative capabilities of large language models, and through coarse-grained filtering of external knowledge base data, efficient semantic enhancement is achieved in the model reasoning process of the threat intelligence question-answering system, thereby improving the real-time performance, accuracy, and interpretability of the generated content. The following description, in conjunction with the accompanying drawings, illustrates the solution described in this application, which includes the following steps: S1: Obtain the input threat intelligence query information and generate a query semantic representation vector.

[0069] Specifically, the system first loads an external threat intelligence database from a pre-defined threat intelligence data source. This external threat intelligence database is a collection of databases storing structured and unstructured threat intelligence, including but not limited to publicly available reports, vulnerability databases, and threat analysis reports. It supports periodic automatic updates and incremental synchronization mechanisms to ensure data timeliness. The system then receives the input query text, which serves as input for subsequent analysis and reasoning.

[0070] The input query text is semantically encoded by mapping the text to a semantic vector space. Its semantic representation can be obtained from the text representation vector generated by a pre-trained language model, i.e., the query semantic representation vector. Simultaneously, the event description text from the external threat intelligence database is vectorized and encoded.

[0071] S2, based on a hybrid retrieval strategy, retrieves related threat events in the semantic representation vector of the external threat intelligence coarse-grained retrieval query, and selects the M texts with the highest comprehensive relevance scores as fine-grained retrieval input, i.e., the candidate intelligence document set.

[0072] Specifically, during the retrieval process, a hybrid retrieval strategy combining semantic relevance calculation and keyword matching is employed. This involves calculating the similarity (e.g., cosine similarity) between the query semantic vector and the semantic vectors of documents in the intelligence database, and combining this with keyword matching scores obtained through methods such as term statistics. These two factors are then weighted and fused to obtain a comprehensive document retrieval score. Finally, based on the comprehensive retrieval score, the external threat intelligence text with the highest Top-M score is selected as the input for subsequent fine-grained knowledge retrieval, i.e., the candidate intelligence document set.

[0073] S3, based on a supervised ranking model, performs fine-grained ranking of external threat intelligence to obtain a set of selected documents, and then uses an attention bias matrix based on the relevance scores of the selected documents.

[0074] Specifically, the Top-M external threat intelligence texts selected in S2 are used as a candidate intelligence document set and input into the trained supervised ranking model. The trained supervised ranking model acquires the ability to score the relevance of candidate documents through training. The supervised ranking model is trained using a mixture of Positive, Negative, and Hard Negative training data. Positive samples are high-scoring, high-quality paragraphs in traditional retrieval; Negative samples are paragraphs with different topics and low scores in traditional retrieval; and Hard Negative samples are paragraphs with high similarity but low relevance. Then, based on the obtained relevance scores of the candidate documents, the Top-N external threat intelligence texts with the highest scores are selected as the refined document set.

[0075] The relevance scores of selected documents in the selected document set are standardized and then mapped to the attention bias in the inference process of the threat intelligence big data model.

[0076] In this embodiment, a threat intelligence knowledge base adapted to the attention bias-enhanced threat intelligence question-answering method is first constructed. The knowledge base covers four types of threat objects: IP, domain name, URL, and vulnerability, totaling 10,080 knowledge entries. Each knowledge entry includes at least the following fields: object identifier, object type, risk level, threat type, evidence summary, and recommended action. These fields are organized in a unified JSONL document format and used as input to an external threat intelligence database. Based on these knowledge entries, an evaluation query set is generated around tasks such as risk analysis, threat type determination, and action recommendations. A total of 30,240 query samples are obtained, used for end-to-end verification of the coarse-grained retrieval, fine-grained ranking, and result generation processes in the method of this invention.

[0077] During the supervised ranking model training phase, a cross-encoder-based supervised ranking model is used to re-rank candidate documents. Training samples consist of Positive, Hard Negative, and Negative samples. Positive samples represent candidate documents that exactly match the query object; Hard Negative samples represent candidate documents of the same type and similar expression as the query object but with different objects; and Negative samples represent cross-type or weakly related documents. To enable the model to simultaneously learn the discriminative relationship between exact matching relationships and highly similar but low-relevance samples, a three-level relevance label is applied to the training samples, mapped to three supervision labels: 2, 1, and 0.

[0078] In this embodiment, the supervised ranking model is trained for one epoch, with a total of 8064 training steps. After training, Pearson correlation coefficient and Spearman correlation coefficient are used on the validation set to evaluate the consistency between the model output score and the standard label, as well as the ranking consistency.

[0079] Table 1 presents the experimental results of the supervised ranking model training. The results show that the trained supervised ranking model can effectively learn the relevance mapping relationship between threat intelligence queries and candidate documents, and can form a highly consistent ranking result.

[0080] Table 1. Training Results of Supervised Ranking Model S4, based on a carefully selected document set, guides large models to perform logically consistent reasoning and generation through a knowledge-driven prompting mechanism and a zero-shot thought chain enhancement strategy.

[0081] Specifically, selected documents are used as the input context for the large language model, and prompt templates incorporating zero-shot thought chains are constructed to guide the generation process. An example is shown below: "Given the following evidence and knowledge path: {Selected Documents}; and for the question: {Input Query}, please analyze the possible threat intelligence types, determine the attacker's identity and corresponding confidence levels step by step, and provide evidence-based defense recommendations. Please clearly indicate the evidence number or knowledge path identifier on which each judgment is based in the output. Hint: Let's think about it step by step." For example, Table 2 provides the results of different schemes to further verify the application effect of the attention bias-enhanced threat intelligence question answering method in the complete threat intelligence question answering chain. This embodiment sets up three sets of comparison schemes: Firstly, the rule-based coarse ranking is used, which involves using a rule-based scoring method to sort the candidate documents.

[0082] Secondly, the supervised ranking model is used for fine ranking, which involves using a trained supervised ranking model to perform rule-based fine ranking and obtain a set of selected documents, while the rest of the process remains unchanged.

[0083] Third, the supervised ranking model fine ranking is combined with zero-thought chain prompt templates, that is, based on the supervised ranking model fine ranking, zero-thought chain prompt templates are introduced and the final result is output.

[0084] For example, in the above experiments, the top three recall rate (Recall@3) and top five recall rate (Recall@5) were used as evaluation metrics to measure whether correct documents were recalled and ranked highly. The generation layer used risk level determination accuracy (RiskAcc) and threat type determination F1 score (JudgmentF1) as evaluation metrics to measure the accuracy of risk level determination and the degree of matching of threat type sets, respectively. Experiments show that in structured threat intelligence question answering scenarios, this attention bias-enhanced threat intelligence question answering method can stably output key information such as risk level and threat type.

[0085] Table 2 Evaluation Results of Threat Intelligence Question Answering Method Based on Attention Bias Enhancement It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0086] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0087] Figure 2 The diagram illustrates the structure of a threat intelligence question-answering device based on attention bias enhancement provided in an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown, and are described in detail below: like Figure 2 As shown, the threat intelligence question-answering device 2 based on attention bias enhancement includes: The input parsing module 201 is used to obtain the input threat intelligence query information and generate a query semantic representation vector.

[0088] The candidate intelligence document filtering module 202 is used to obtain a set of candidate intelligence documents from an external threat intelligence database by using a combination of semantic matching and keyword matching based on the query semantic representation vector.

[0089] The selected document filtering module 203 is used to input the candidate intelligence document set into the trained supervised ranking model, output the relevance score of each candidate document, and filter out the selected document set based on the relevance score.

[0090] The large language model improvement module 204 is used to convert the relevance scores of selected documents in the selected document set into word-level attention bias matrices, which are then superimposed on the original attention scores of the large language model to obtain an improved large language model.

[0091] The answer generation module 205 is used to improve the large language model by taking each selected document as context input and using a zero-shot chain-like thinking prompting strategy to guide the improved large language model to generate answers.

[0092] Figure 3 This is a schematic diagram of the threat intelligence question-answering device based on attention bias enhancement provided in an embodiment of the present invention. Figure 3 As shown, the threat intelligence question-answering device 3 based on attention bias enhancement in this embodiment includes a processor 300 and a memory 301. The memory 301 stores a computer program 302. When the processor 300 executes the computer program 302, it implements the steps in the various method embodiments described above. Alternatively, when the processor 300 executes the computer program 302, it implements the functions of each module / unit in the various device embodiments described above.

[0093] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0094] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A threat intelligence question answering method based on attention bias enhancement, characterized in that, include: Obtain the input threat intelligence query information and generate a query semantic representation vector; Based on the query semantic representation vector, a candidate intelligence document set is obtained from an external threat intelligence database through a hybrid retrieval method that combines semantic matching and keyword matching. The candidate intelligence document set is input into the trained supervised ranking model, which outputs the relevance score of each candidate document, and a selected document set is selected based on the relevance score. The relevance scores of the selected documents in the selected document set are converted into word-level attention bias matrices, which are then superimposed on the original attention scores of the large language model to obtain an improved large language model. Each selected document is used as context input to the improved large language model, and a zero-shot chain-like thinking prompting strategy is adopted to guide the improved large language model to generate answers.

2. The threat intelligence question answering method based on attention bias enhancement according to claim 1, characterized in that, Based on the query semantic representation vector, a candidate intelligence document set is obtained from an external threat intelligence database through a hybrid retrieval method combining semantic matching and keyword matching, including: Semantic encoding is performed on each intelligence document in the external threat intelligence database to generate a semantic representation vector for each intelligence document; Calculate the similarity between the query semantic representation vector and the semantic representation vector of each intelligence document to obtain the semantic matching score of each intelligence document; Based on the keywords in the threat intelligence query information, the keyword matching score of each intelligence document is calculated using the term matching method. For any intelligence document, the keyword matching score and the semantic matching score of the intelligence document are weighted and fused to obtain the comprehensive retrieval score of the intelligence document. Based on the comprehensive retrieval score of each intelligence document, the intelligence documents are sorted in descending order, and the first first preset number of intelligence documents are selected to form the candidate intelligence document set.

3. The threat intelligence question answering method based on attention bias enhancement according to claim 1, characterized in that, Before inputting the candidate intelligence document set into the trained supervised ranking model, the process also includes: Obtain a training dataset, wherein the training samples in the training dataset include labeled query information, positive sample documents, negative sample documents, and semantically related but actually irrelevant sample documents; Construct a cross-coding sorting model; Using the training dataset, the cross-coding ranking model is trained to obtain the trained supervised ranking model; The loss function includes a sorting cross-entropy loss term and an interval loss term. The sorting cross-entropy loss term is used to increase the sorting probability of the positive sample documents, and the interval loss term is used to force the relevance score of the positive sample documents to be higher than the relevance score of the semantically related but actually unrelated sample documents by an interval threshold.

4. The threat intelligence question answering method based on attention bias enhancement according to claim 3, characterized in that, The formula for calculating the loss function is as follows: ； in, Let the loss function be... The sorting cross-entropy loss term is... For the interval loss term, The relevance score of the positive sample document. For the first The relevance score of each of the negative sample documents. For the first The relevance score of the semantically related but actually irrelevant sample documents. To lose weight, The interval threshold is... It is an exponential function. It is a function for maximizing the value.

5. The threat intelligence question answering method based on attention bias enhancement according to claim 3, characterized in that, The candidate intelligence document set is input into the trained supervised ranking model, and the relevance scores of each candidate document are output to rank the candidate documents, thus selecting a set of premium documents, including: For any candidate document in the candidate intelligence document set, the query semantic representation vector is concatenated with the candidate document to construct the input sequence corresponding to the candidate document; Each input sequence is input into the trained supervised ranking model, which outputs a relevance score for each candidate document. The candidate documents are sorted in descending order based on their relevance scores. The first second preset number of candidate documents are selected to form the selected document set.

6. The threat intelligence question answering method based on attention bias enhancement according to claim 1, characterized in that, The relevance scores of the selected documents in the selected document set are converted into word-level attention bias matrices, which are then superimposed on the original attention scores of the large language model to obtain an improved large language model, including: For any selected document in the selected document set, the relevance score of the selected document is standardized and mapped to the document-level attention bias coefficient of the selected document. Each document-level attention bias coefficient is extended to each word of the corresponding selected document to obtain a word-level attention bias matrix; In each attention layer of the large language model, the word-level attention bias matrix is ​​superimposed on the original attention score to obtain the biased attention score. The output of each attention layer is calculated based on the biased attention score to obtain the improved large language model.

7. The threat intelligence question answering method based on attention bias enhancement according to claim 6, characterized in that, The formula for calculating the word-level attention bias matrix is ​​as follows: ； ； in, The word-level attention bias matrix The Middle Line 1 q Column elements, as a word element q The document-level bias coefficient corresponding to the document. For the first The document-level attention bias coefficient of each of the selected documents. The bias strength coefficient, For the first The relevance score of the selected documents The mean of all the relevance scores in the selected document set. The standard deviation of all the relevance scores in the selected document set.

8. The threat intelligence question answering method based on attention bias enhancement according to claim 6, characterized in that, The formula for calculating the biased attention score is as follows: ； in, The biased attention score. For normalization function, For querying the matrix, The key matrix, For value matrices, The feature dimension corresponding to the attention head. is the word-level attention bias matrix.

9. A threat intelligence question-answering device based on attention bias enhancement, characterized in that, include: The input parsing module is used to obtain the input threat intelligence query information and generate a query semantic representation vector; The candidate intelligence document filtering module is used to obtain a set of candidate intelligence documents from an external threat intelligence database by performing a hybrid retrieval of semantic matching and keyword matching based on the query semantic representation vector. The selected document filtering module is used to input the candidate intelligence document set into the trained supervised ranking model, output the relevance score of each candidate document, and filter out the selected document set based on the relevance score. The large language model improvement module is used to convert the relevance scores of the selected documents in the selected document set into a word-level attention bias matrix, which is then superimposed on the original attention score of the large language model to obtain an improved large language model. The answer generation module is used to input various selected documents as context into the improved large language model, and adopts a zero-shot chain-like thinking prompting strategy to guide the improved large language model to generate answers.

10. A threat intelligence question-answering device based on attention bias enhancement, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the threat intelligence question answering method based on attention bias enhancement as described in any one of claims 1 to 8.

Images