Threat intelligence entity detection method based on BERT and deep learning model

Abstract

The present invention relates to a threat intelligence entity detection method based on BERT and deep learning model, comprising: first, using a pre-trained BERT model to capture the basic semantic information of a text and construct a grammatical structure diagram; then, the grammatical structure diagram is sent to a graph attention network for processing to analyze the complex dependencies between entities; at the same time, the CLS vector output by the BERT model is spliced ​​with the global vector obtained by Text‑CNN processing to form an HCV containing global context information and local detail features; in addition, word temporal context information and the importance association between entity words are obtained; finally, vectors from different modules are fused and placed in a conditional random field layer for entity recognition to obtain the output of threat entities. The present invention exhibits better performance when processing professional terms and complex contexts in the field of network security.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and specifically relates to a threat intelligence entity detection method based on BERT and a deep learning model. Background Art

[0002] In this fast-paced digital age, we have witnessed a rapid increase in complex cyber attacks such as 0-day attacks and advanced persistent threats (APTs). These attacks can often easily bypass traditional defense measures and invade critical infrastructure, causing serious consequences. In order to deal with these increasingly complex threats, security experts have proposed CTI reports, which not only bring together a series of indicators of compromise (IOCs), but also describe in detail in natural language the sequence of attacks, the impact on the attacked system, information about the attacker's organization, and many other aspects. Unlike well-known security databases such as CVE and ExploitDB, CTI provides more comprehensive and valuable early warning information, including malicious IP addresses, domain names, malware and their attack patterns, enabling organizations to respond quickly when encountering suspicious external or internal threats. CTI is usually released in various forms such as technical reports, white papers, blogs, and newsgroups, and is regarded as an effective means to combat advanced threats such as APTs, helping security experts quickly identify attack steps and develop effective defense measures. However, the massive growth of cyber threat reports has challenged the analytical capabilities of security experts, and more efficient processing and analysis methods are urgently needed.

[0003] In the field of network security, the complexity of threat intelligence processing is not only due to the huge amount of information, but also due to the high density of professional terms and their ambiguity in different contexts. For example, "OceanLotus" usually refers to a plant in a conventional context, but in the field of network security, it actually refers to a well-known APT organization - OceanLotus (APT32). This context-dependent ambiguity makes it difficult for automated tools to accurately identify and classify related entities without sufficient context understanding capabilities. Existing threat indicator (IOC) extraction methods, such as CleanMX, PhishTank, IOC Finder, and Gartner peer insight, mainly rely on a set of regular expressions to identify specific types of IOCs. Although this method is direct, it is mainly based on dictionary matching and lacks understanding of context semantics, which can easily lead to the omission of key threat objects. In addition, due to the lack of a full description of the threat event, isolated IOCs are difficult to provide CTI users with a comprehensive understanding of upcoming threats. On the other hand, although deep learning frameworks such as HINTI and EXTRACTOR can be used to extract threat intelligence from unstructured text, these methods are relatively simple and have limited effects in extracting named entities (NER). Although the current more advanced NER models, such as MHSA and GPT, have excellent performance in many fields, they were not originally designed to process threat intelligence. Therefore, these models face challenges in understanding professional terms and complex contexts in the field of network security, and it is difficult to achieve good generalization effects in the field of CTI. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a threat intelligence entity detection method based on Bert and deep learning model. On the basis of the traditional deep learning framework, the present invention introduces graph attention network and Hybrid Context Vector, aiming to enhance the accuracy and efficiency of automatic identification of threat entities in network threat intelligence text.

[0005] Terminology explanation:

[0006] 1. BERT model, BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. It captures contextual relationships by performing bidirectional training on a large amount of text data. It is suitable for various NLP tasks such as question answering and text classification.

[0007] 2. Text-CNN. Text-CNN is a convolutional neural network (CNN) model applied to text classification tasks. It extracts n-gram features from text for classification tasks by applying convolution kernels to word vectors of text.

[0008] 3. Global Text CNN layer. The Global Text CNN layer is a variant of Text-CNN. It usually extracts global context features by applying convolution kernels on the entire text to improve the understanding of long texts.

[0009] 4. Hybrid Context Vector: Hybrid Context Vector refers to a hybrid context vector formed by combining multiple context information (in this article, it is a new vector obtained by combining the CLS vector generated by Text-CNN and BERT), which enhances the model's understanding of text semantics.

[0010] 5. BiLSTM Layer, BiLSTM (Bidirectional Long Short-Term Memory) layer is a bidirectional long short-term memory network layer, which can simultaneously consider the contextual information of the input sequence and process the sequence through two LSTM networks (one from front to back and the other from back to front) to capture more comprehensive contextual information.

[0011] 6. Attention Layer: Attention Layer is a mechanism that can dynamically assign attention weights, allowing the model to focus on the parts related to the current task when processing the input sequence, thereby improving the model's ability to extract important features.

[0012] 7. Bidirectional long short-term memory network BiLSTM Layer, that is, BiLSTM Layer.

[0013] 8. Multi-head attention mechanism Attention Layer, that is, Attention Layer.

[0014] According to some embodiments, a first solution of the present invention provides a threat intelligence entity detection method based on Bert and a deep learning model, which adopts the following technical solution:

[0015] The threat intelligence entity detection method based on BERT and deep learning model includes:

[0016] First, the pre-trained BERT model is used to capture the basic semantic information of the text, and a grammatical structure diagram is constructed based on the basic semantic information of the text;

[0017] Then, the grammatical structure graph is sent to the graph attention network (GAT layer) for processing to analyze the complex dependencies between entities. The BERT model is used to generate an embedding vector, which is processed by Text-CNN to obtain a global vector. The CLS vector output by the BERT model is concatenated with the global vector to form a HybridContext Vector (HCV) containing global context information and local detail features. In addition, the embedding vector generated by the BERT model is sent to the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words.

[0018] Finally, the vectors from different modules are fused and put into the conditional random field (CRF) layer for entity recognition to obtain the output of threat entities.

[0019] Preferably, according to the present invention, the BERT model outputs a CLS vector and a basic semantic vector H containing basic semantic information; including:

[0020] Define the input text X as X = x1, x2, x3, ..., x i , …, x n , where x i is the i-th word in the text X, and n is the length of the sentence. The input text X is fed into the BERT model for embedding processing to obtain a vector sequence:

[0021] H={h1,h2,h3,…,h i ,…,h n};

[0022] Among them, h i For word x i The vector embedding of dm is the embedding dimension representation of the vector; at the same time, the CLS vector is obtained, where the CLS vector

[0023] Preferably, according to the present invention, a global vector is obtained by Text-CNN processing; comprising:

[0024] Text-CNN extracts key local features from different context windows by applying multiple convolution kernels. These local features are aggregated into a global vector. The vector sequence H is passed to a Global Text CNN layer, and the global vector h of the entire sentence is extracted through the convolution and pooling operations of the GlobalText CNN layer. global :

[0025]

[0026] Among them, TextCNN() inputs the basic semantic vector H obtained after processing the BERT model into Text-CNN to obtain the global vector h global .

[0027] According to the preferred embodiment of the present invention, the CLS vector h output by the BERT model is CLs And the global vector h obtained by Text-CNN processing global Concatenate them to form a Hybrid ContextVector (HCV) containing global context information and local detail features; as shown below:

[0028] HCV=[h CLS ;h global ]

[0029] Among them, ";" represents the concatenation operation of vectors.

[0030] Preferably, according to the present invention, a grammatical structure diagram is constructed based on basic semantic information of the text; including:

[0031] Convert input text into a structured syntax tree;

[0032] According to the structural syntax tree, the adjacency matrix A is constructed to represent the grammatical dependencies between words, as shown below:

[0033]

[0034] A ij Represents an element of the adjacency matrix A, used to represent word x i 、x j The grammatical relationship between i represents the i-th word in the text, x j Represents the jth word in the input text;

[0035] Construct a grammatical structure graph G = (H, A) as the input of the graph attention network, where H is the word vector set of the output sentence after being processed by the BERT model, and A is the adjacency matrix that describes the dependencies between words.

[0036] Preferably, according to the present invention, the grammatical structure graph is sent to the graph attention network for processing to analyze the complex dependency relationship between entities; including:

[0037] The grammatical structure graph G is input into the graph attention network, which updates the representation of each node through the following formula; the processing of the graph attention network is described as:

[0038]

[0039] Among them, h′ i represents the new representation of node i, W is the learnable weight matrix, α ij is the attention coefficient between nodes i and j, represents the set of neighbor nodes of node i, σ is a nonlinear activation function, a is a learnable vector, || represents the vector concatenation operation, LeakyReLU is the ReLU activation function with leakage, and exp() is the abbreviation of the exponential function;

[0040] After being processed by the graph attention network, the vector representation of each word is updated to form a new sentence representation, as shown below:

[0041] H′={h′1, h′2,..., h′ i , ..., h′ n}

[0042] in, dm is the embedding dimension representation of the vector, H′ is the new vector sequence after being processed by the graph attention network, and each h′ i Both incorporate information from adjacent words.

[0043] Preferably, according to the present invention, the embedding vectors generated by the BERT model are respectively fed into the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words; including:

[0044] Input the vector representation H of the sentence into the bidirectional long short-term memory network BiLSTM Layer; as shown below:

[0045]

[0046] Among them, h i is the embedding representation of the i-th word in the sentence, and It is the bidirectional representation of the word vector after being processed by the bidirectional long short-term memory network BiLSTMLayer. Combines information from the previous and following contexts of the sentence; BiLSTM() means inputting the vector representation H of the input sentence into the bidirectional LSTM network and outputting the updated hidden state; by concatenating the forward and reverse outputs of the bidirectional long short-term memory network BiLSTM Layer, a bidirectional representation of each word is obtained, thus forming a new representation of the sentence:

[0047]

[0048] in, is the i-th word vector representation, Reflects the semantic position and contextual relationship of the word in the whole sentence;

[0049] Apply multi-head attention mechanism; including:

[0050] Through different weight matrices W Q , W K , W V and bias b Q 、b K 、b V To transform the original embedding matrix H, we get three new matrices H Q , H K , H V , representing queries, keys, and values ​​respectively:

[0051] H Q =W Q ·H+b Q

[0052] H K =W K ·H+b K

[0053] H V =W V ·H+b V

[0054] in, b Q , b K , b V are the bias terms, The new matrix H Q , H K , H V It is used to calculate the attention score A. The formula of the self-attention mechanism is as follows:

[0055]

[0056] According to the attention score A and the value matrix H V Calculate the attention vector sequence H att :

[0057] H att =Attention(Q,K,V)=A·H V

[0058] entire Provides an optimized sentence representation where each Represents the word vector after weighting by the attention mechanism;

[0059] entire

[0060] Preferably, according to the present invention, vectors from different modules are fused; including:

[0061] The vectors from the multi-head attention mechanism Attention Layer, the bidirectional long short-term memory network BiLSTM Layer, the graph attention network, and the Hybrid Context Vector are concatenated. For each word in the sentence, its comprehensive feature vector The definition is as follows:

[0062]

[0063] Among them, each The whole sentence Where n is the length of the sentence.

[0064] Preferably, according to the present invention, a conditional random field (CRF) layer is used to identify the entity to obtain a detection result, specifically:

[0065] A linear transformation layer is introduced, whose operation is expressed as:

[0066] S=H c m b ×W+b

[0067] Where W is a 6d dimension m ×Lsize weight matrix, Lsize is the number of labels, b is the bias vector;

[0068] The mapped score matrix S is input to the conditional random field layer. Each row of the score matrix S represents the score of a word on all labels. The conditional random field layer uses the score matrix S and the transition probability between labels to calculate the most likely label sequence:

[0069]

[0070] Where Ψ represents the score function for extracting a specific label sequence from the score matrix S, y is the entity label sequence, Y is the set of all possible label sequences, S i is the label score vector corresponding to the i-th word; through the calculation of the conditional random field layer, the entity label of each word is marked, and each entity and entity label are output.

[0071] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a threat intelligence entity detection method based on BERT and a deep learning model are implemented.

[0072] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a threat intelligence entity detection method based on BERT and a deep learning model.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] The present invention proposes a new named entity recognition (NER) model, GAT-TI, which is optimized specifically for the specific needs of the field of cybersecurity threat intelligence (CTI). By introducing the graph attention network (GAT) and Hybrid ContextVector, GAT-TI can more accurately capture the complex dependencies between entities in CTI text, significantly improving the accuracy of entity recognition. Compared with the traditional NER model, the present invention shows better performance when processing professional terms and complex contexts in the field of cybersecurity. The present invention uses the pre-trained BERT model to capture the basic semantic information of the text, and combines the graph attention network to construct a grammatical structure graph to accurately analyze the dependencies between entities. At the same time, the Hybrid Context Vector combines global context information and local detail features, greatly enriching the model's understanding depth of text and improving its ability to handle complex threat intelligence. GAT-TI achieves a more robust clustering effect through a supervised contrastive learning method. Supervised contrastive learning encourages the encoder to provide tightly aligned representations for samples in the same class, reduces the performance fluctuation of the model on different data sets, and improves the generalization ability and classification accuracy of the model. In view of the high density and ambiguity of professional terms in the CTI field, the present invention introduces a multi-head attention mechanism and a BiLSTM layer to capture the temporal and contextual information in the text, effectively handle complex contexts and ambiguity problems, and improve the accuracy and recall of entity recognition in threat intelligence text. The present invention combines the outputs of multiple deep learning modules, performs entity recognition and classification through a linear transformation layer and a conditional random field (CRF) layer, and ensures accurate conversion from text to entity labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0076] Figure 1 It is a flowchart of a threat intelligence entity detection method based on Bert and deep learning model in an embodiment of the present invention;

[0077] Figure 2 Schematic diagram of the comparison of F1 performance of different models for some entities in CDTI. DETAILED DESCRIPTION

[0078] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0079] Example 1

[0080] Threat intelligence entity detection method based on BERT and deep learning model, such as Figure 1 As shown, including:

[0081] First, the pre-trained BERT model is used to capture the basic semantic information of the text, and a grammatical structure diagram is constructed based on the basic semantic information of the text;

[0082] Then, the grammatical structure graph is sent to the graph attention network (GAT layer) for processing to analyze the complex dependencies between entities. The BERT model is used to generate an embedding vector, which is processed by Text-CNN to obtain a global vector. The CLS vector output by the BERT model is concatenated with the global vector to form a HybridContext Vector (HCV) containing global context information and local detail features. In addition, the embedding vector generated by the BERT model is sent to the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words.

[0083] Finally, the vectors from different modules are fused and put into the conditional random field (CRF) layer for entity recognition to obtain the output of threat entities.

[0084] Example 2

[0085] The threat intelligence entity detection method based on Bert and deep learning model described in Example 1 is different in that:

[0086] The BERT model outputs a CLS vector and a basic semantic vector H containing basic semantic information; including:

[0087] Define the input text X as X = x1, x2, x3, ..., x i , …, x n , where x i is the i-th word in the text X, and n is the length of the sentence. The input text X is fed into the BERT model for embedding processing to obtain a vector sequence:

[0088] H={h1,h2,h3,…,h i ,…,h n};

[0089] Among them, h i For word xi The vector embedding of d m is the embedding dimension representation of the vector; at the same time, the CLS vector is obtained, where the CLS vector By using the BERT model, a deep semantic representation of each word in the text is obtained, where the CLS vector serves as a compressed representation of the entire text, capturing the global semantics of the text.

[0090] The global vector is obtained through Text-CNN processing; including:

[0091] In order to enhance the BERT model's ability to capture local text features, Text-CNN was introduced. Text-CNN extracts key local features from different context windows by applying multiple convolution kernels. These local features are aggregated into a global vector. The vector sequence H is passed to a Global Text CNN layer, and the global vector h of the entire sentence is extracted through the convolution and pooling operations of the Global Text CNN layer. global :

[0092]

[0093] Among them, TextCNN() inputs the basic semantic vector H obtained after processing the BERT model into Text-CNN to obtain the global vector h global .

[0094] The CLS vector h output by the BERT model CLs And the global vector h obtained by Text-CNN processing global Concatenate to form a Hybrid Context Vector (HCV) containing global context information and local detail features; as shown below:

[0095] By fusing the CLS vector output by the BERT model and the global vector obtained through Text-CNN processing, a novel vector representation, Hybrid Context Vector (HCV), is constructed. This vector effectively combines global context information and local detail features, providing a more powerful feature basis for entity recognition in threat intelligence. This vector can effectively improve the accuracy of entity recognition.

[0096] HCV=[h CLS ;h global ]

[0097] Among them, ";" represents the concatenation operation of vectors.

[0098] Construct a grammatical structure diagram based on the basic semantic information of the text; including:

[0099] Convert input text into a structured syntax tree;

[0100] According to the structural syntax tree, the adjacency matrix A is constructed to represent the grammatical dependencies between words, as shown below:

[0101]

[0102] A ij Represents an element of the adjacency matrix A, used to represent word x i 、x j The grammatical relationship between i represents the i-th word in the text, x j Represents the jth word in the input text;

[0103] This adjacency matrix A captures the structural relationship between words in the sentence and provides a basis for graph construction. With the word vectors and adjacency matrix, a grammatical structure graph G = (H, A) is constructed as the input of the graph attention network, where H is the word vector set of the output sentence after being processed by the BERT model, and A is the adjacency matrix that describes the dependencies between words.

[0104] The grammatical structure graph is fed into the graph attention network to analyze the complex dependencies between entities, including:

[0105] We use natural language processing tools to conduct in-depth grammatical structure analysis of texts, and use the Graph Attention Network (GAT) to significantly enhance the feature expression of text data. Specifically:

[0106] This process not only involves accurate part-of-speech tagging of the text, but also includes building a complex adjacency matrix that reflects the grammatical dependencies between words. By comprehensively utilizing the information of text sequences and grammatical structure graphs, the accuracy of the model in identifying threat intelligence entities in complex security reports is significantly improved.

[0107] The grammatical structure graph G is input into the graph attention network, which updates the representation of each node through the following formula to accurately capture and strengthen the complex relationship between entities. The processing of the graph attention network is described as:

[0108]

[0109] Among them, h′ i represents the new representation of node i, W is the learnable weight matrix, α ij is the attention coefficient between nodes i and j, represents the set of neighbor nodes of node i, σ is a nonlinear activation function, a is a learnable vector, || represents the vector concatenation operation, LeakyReLU is the ReLU activation function with leakage, and exp() is the abbreviation of the exponential function; this mechanism enables the network to pay attention to other words that are grammatically or semantically related to the current word, thereby obtaining a richer and more discriminative feature representation.

[0110] After being processed by the graph attention network, the vector representation of each word is updated to form a new sentence representation, as shown below:

[0111] H′={h′1, h′2, .., h′ i , ..., h′ n}

[0112] in, d m is the embedding dimension representation of the vector, H′ is the new vector sequence after being processed by the graph attention network, and each h′ i Both incorporate information from adjacent words, enhancing the model's understanding of the entire sentence structure and semantics.

[0113] The embedding vectors generated by the BERT model are fed into the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words; including:

[0114] In order to capture the word timing and context information, the vector representation H of the sentence is input into the bidirectional long short-term memory network BiLSTM Layer; the bidirectional long short-term memory network BiLSTM Layer is a network structure specially designed to process sequence data, which can effectively capture the characteristics of forward and backward transmission of information in the time series: as shown below:

[0115]

[0116] Among them, h i is the embedding representation of the i-th word in the sentence, and It is the bidirectional representation of the word vector after being processed by the bidirectional long short-term memory network BiLSTMLayer. Combines information from the previous and following contexts of the sentence; BiLSTM() means inputting the vector representation H of the input sentence into the bidirectional LSTM network and outputting the updated hidden state; by concatenating the forward and reverse outputs of the bidirectional long short-term memory network BiLSTM Layer, a bidirectional representation of each word is obtained, thus forming a new representation of the sentence:

[0117]

[0118] in, is the i-th word vector representation, Reflects the semantic position and contextual relationship of the word in the whole sentence;

[0119] In order to further refine the subtle connections between entities in the text, a multi-head attention mechanism is applied; including:

[0120] Through different weight matrices W Q , W K , W V and bias b Q 、b K 、b V To transform the original embedding matrix H, we get three new matrices H Q , H K , H V , representing queries, keys, and values ​​respectively:

[0121] H Q =W Q ·H+b Q

[0122] H K =W K ·H+b K

[0123] H V =W V ·H+b V

[0124] in, b Q , b K , b V are the bias terms, The new matrix H Q , H K , H V It is used to calculate the attention score A. The formula of the self-attention mechanism is as follows:

[0125]

[0126] According to the attention score A and the value matrix H V Calculate the attention vector sequence H att :

[0127] H att =Attention(Q,K,V)=A·H V

[0128] entire Provides an optimized sentence representation where each Represents the word vector after weighting by the attention mechanism;

[0129] entire It effectively enhances the model's understanding of key entities and their contextual relationships.

[0130] H att It represents the sequence of attention vectors calculated by the multi-head attention mechanism, which is a new representation after weighting the original sentence representation.

[0131] Fusion of vectors from different modules; including:

[0132] In order to make full use of the outputs of multiple deep learning modules, the vectors from the multi-head attention mechanism AttentionLayer, the bidirectional long short-term memory network BiLSTM Layer, the graph attention network and the Hybrid Context Vector are concatenated to construct a richer feature representation. For each word in the sentence, its comprehensive feature vector The definition is as follows:

[0133]

[0134] Among them, each The whole sentence Where n is the length of the sentence.

[0135] The conditional random field (CRF) layer is used to identify entities and obtain the detection results, which are as follows:

[0136] The core of this process is to integrate the information provided by each module to ensure that each entity is accurately identified and output in a predetermined format. By introducing the linear transformation layer and the CRF layer, the model can map the comprehensive feature vector to a space that matches the number of entity tags and use the structured prediction ability of the CRF layer to accurately annotate the entities in the sentence. This strategy ensures the accuracy and completeness of capturing and outputting key entities from the input text.

[0137] In order to map the comprehensive feature vector to a space matching the number of entity labels, a linear transformation layer is introduced, whose operation is expressed as:

[0138] S=H c m b ×W+b

[0139] Where W is a 6d dimension m×Lsize weight matrix, Lsize is the number of tags, b is the bias vector; this step maps the comprehensive feature vector of each word to a new space, where each dimension corresponds to the score of a possible tag.

[0140] The mapped score matrix S is input to the conditional random field layer. Each row of the score matrix S represents the score of a word on all labels. The conditional random field layer uses the score matrix S and the transition probability between labels to calculate the most likely label sequence:

[0141]

[0142] Where Ψ represents the score function for extracting a specific label sequence from the score matrix S, y is the entity label sequence, Y is the set of all possible label sequences, S i is the label score vector corresponding to the i-th word; through the calculation of the conditional random field layer, the entity label of each word is accurately marked, and each entity and entity label are output.

[0143] In this embodiment, the data sets used are: For the data set, two public data sets, FeedRef2022 and CyNER, are used. In addition, in order to better evaluate the method of the present invention, this embodiment uses a web crawler to crawl APT threat intelligence reports regularly released by more than 20 network security vendors including RecordedFuture, Kaspersky, FireEye, etc. After obtaining the threat report, PDFMiner (a format conversion tool) is used to convert the report in PDF format into text format, and special symbols and advertising information in the threat report are removed, leaving only the main text of the report. Finally, the annotation tool brat is used to manually annotate the threat intelligence text using the BIOES notation method to generate a Custom Threat Intelligence Dataset (CTID).

[0144] Compare the model to the baseline:

[0145] (Lample et al. 2016) is a classic neural model for planar NER, which utilizes an LSTM-CRF neural network.

[0146] (MS AI-Qurishi et al. 2021)MS Al-Qurishi et al. adopted an architecture combining BERT, fully connected layers and conditional random fields (CRF) to handle flat named entity recognition (NER) tasks, showing good results.

[0147] (Hyeonseong Joa et al. 2022) is a model in the field of threat intelligence. The NER model in the paper uses BERT-BiLSTM-CRF to process threat intelligence sentences, and the model is well adapted to the professional terminology and complex context in the field of threat intelligence.

[0148] The effect data of different models on different data sets are shown in Table 1:

[0149] Table 1

[0150]

[0151] In order to compare the performance of different models in extracting different entities, we selected the three entities with the highest frequency and the two entities with the lowest frequency in the CDTI dataset and calculated the F1 scores of these models. Figure 2 Through this comparison, we can clearly see how each model performs in processing entities of different frequencies, and then evaluate their specific effects in the entity extraction task.

[0152] Example 3

[0153] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the threat intelligence entity detection method based on Bert and deep learning model described in Example 1 or 2 are implemented.

[0154] Example 4

[0155] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the threat intelligence entity detection method based on Bert and a deep learning model described in Example 1 or 2.

Claims

1. A threat intelligence entity detection method based on BERT and deep learning model, characterized in that: include: First, the pre-trained BERT model is used to capture the basic semantic information of the text, and a grammatical structure diagram is constructed based on the basic semantic information of the text; Then, the grammatical structure graph is sent to the graph attention network for processing to analyze the complex dependencies between entities. The BERT model is used to generate an embedding vector, which is processed by Text-CNN to obtain a global vector. The CLS vector output by the BERT model is concatenated with the global vector to form a Hybrid Context Vector containing global context information and local detail features. In addition, the embedding vector generated by the BERT model is sent to the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words. Finally, the vectors from different modules are fused and put into the conditional random field layer for entity recognition to obtain the output of threat entities; The BERT model outputs a CLS vector and a basic semantic vector H containing basic semantic information; including: Define the input text X as X = x1, x2, x3, ..., x i , …, x n , where x i is the i-th word in the text X, n is the length of the sentence, and the input text X is fed into the BERT model for embedding processing to obtain a vector sequence: H={h1,h2,h3,…,h i ,…,h n }; Among them, h i For word x i The vector embedding of d m is the embedding dimension representation of the vector; at the same time, the CLS vector is obtained, where the CLS vector 2. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: The global vector is obtained through Text-CNN processing; including: Text-CNN extracts key local features from different context windows by applying multiple convolution kernels. These local features are aggregated into a global vector. The vector sequence H is passed to a GlobalText CNN layer, and the global vector h of the entire sentence is extracted through the convolution and pooling operations of the GlobalTextCNN layer. global : Among them, TextCNN() inputs the basic semantic vector H obtained after processing the BERT model into Text-CNN to obtain the global vector h global .

3. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: The CLS vector h output by the BERT model CLS And the global vector h obtained by Text-CNN processing global Concatenate them to form a Hybrid Context Vector containing global context information and local detail features; as shown below: HCV=[h CLS ;h global ] Among them, ";" represents the concatenation operation of vectors.

4. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: Construct a grammatical structure diagram based on the basic semantic information of the text; including: Convert input text into a structured syntax tree; According to the structural syntax tree, the adjacency matrix A is constructed to represent the grammatical dependencies between words, as shown below: A ij Represents an element of the adjacency matrix A, used to represent word x i 、x j The grammatical relationship between i represents the i-th word in the text, x j Represents the jth word in the input text; Construct a grammatical structure graph G = (H, A) as the input of the graph attention network, where H is the word vector set of the output sentence after being processed by the BERT model, and A is the adjacency matrix that describes the dependencies between words.

5. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: The grammatical structure graph is fed into the graph attention network to analyze the complex dependencies between entities, including: The grammatical structure graph G is input into the graph attention network, which updates the representation of each node through the following formula; the processing of the graph attention network is described as: Among them, h′ i represents the new representation of node i, W is the learnable weight matrix, α ij is the attention coefficient between nodes i and j, represents the set of neighbor nodes of node i, σ is a nonlinear activation function, a is a learnable vector, || represents the vector concatenation operation, LeakyReLU is the ReLU activation function with leakage, and exp() is the abbreviation of the exponential function; After being processed by the graph attention network, the vector representation of each word is updated to form a new sentence representation, as shown below: H′={h′1,h′2,...,h′ i ,...,h′ n } in, d m is the embedding dimension representation of the vector, H′ is the new vector sequence after being processed by the graph attention network, and each h′ i Both incorporate information from adjacent words.

6. The threat intelligence entity detection method based on Bert and deep learning model according to claim 5 is characterized in that: The embedding vectors generated by the BERT model are fed into the bidirectional long short-term memory network BiLSTM Layer and the multi-head attention mechanism Attention Layer to obtain the word temporal context information and the importance association between entity words; including: Input the vector representation H of the sentence into the bidirectional long short-term memory network BiLSTM Layer; as shown below: Among them, h i is the embedding representation of the i-th word in the sentence, and It is the bidirectional representation of the word vector after being processed by the bidirectional long short-term memory network BiLSTM Layer. Combines information from the previous and following contexts of the sentence; BiLSTM() means inputting the vector representation H of the input sentence into the bidirectional LSTM network and outputting the updated hidden state; by concatenating the forward and reverse outputs of the bidirectional long short-term memory network BiLSTM Layer, a bidirectional representation of each word is obtained, thus forming a new representation of the sentence: in, is the i-th word vector representation, Reflects the semantic position and contextual relationship of the word in the whole sentence; Apply multi-head attention mechanism; including: Through different weight matrices W Q , W K , W V and bias b Q 、b K 、b V To transform the original embedding matrix H, we get three new matrices H Q , H K , H V , representing query, key, and value respectively: H Q =W Q ·H+b Q H K =W K ·H+b K H V =W V ·H+b V in, b Q , b K , b V are the bias term, H Q , H K , The new matrix H Q , H K , H V It is used to calculate the attention score A. The formula of the self-attention mechanism is as follows: According to the attention score A and the value matrix H V Calculate the attention vector sequence H att : H att =Attention(Q,K,V)=A·H V entire Provides an optimized sentence representation where each Represents the word vector after weighting by the attention mechanism; entire 7. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: Fusion of vectors from different modules; including: The vectors from the multi-head attention mechanism Attention Layer, the bidirectional long short-term memory network BiLSTM Layer, the graph attention network, and the Hybrid Context Vector are concatenated. For each word in the sentence, its comprehensive feature vector The definition is as follows: Among them, each The whole sentence Where n is the length of the sentence.

8. The threat intelligence entity detection method based on Bert and deep learning model according to claim 1 is characterized in that: The conditional random field layer is used to identify entities and obtain the detection results, which are as follows: A linear transformation layer is introduced, whose operation is expressed as: S=H cmb ×W+b Where W is a 6d dimension m ×Lsize weight matrix, Lsize is the number of labels, b is the bias vector; The mapped score matrix S is input to the conditional random field layer. Each row of the score matrix S represents the score of a word on all labels. The conditional random field layer uses the score matrix S and the transition probability between labels to calculate the most likely label sequence: Where Ψ represents the score function for extracting a specific label sequence from the score matrix S, y is the entity label sequence, Y is the set of all possible label sequences, and S i is the label score vector corresponding to the i-th word; through the calculation of the conditional random field layer, the entity label of each word is marked, and each entity and entity label are output.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the threat intelligence entity detection method based on Bert and deep learning model described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the threat intelligence entity detection method based on Bert and deep learning model described in any one of claims 1-8 are implemented.

Images