A named entity recognition method for information security field
By applying fine-tuned BERT model and conditional random field in the field of information security, combined with multi-layer Transformer and attention mechanism, the complexity and ambiguity problems in information security naming entity recognition are solved, and more accurate and comprehensive feature extraction is achieved. , improves the effect of entity recognition.
Patent Information
- Application Number
- CN202111125595.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-09-26
AI Technical Summary
The recognition of named entities in the field of information security faces problems such as complex and diverse entity types, continuous emergence of new creative entities, mixed language and abbreviation semantics, and multiple meanings of one word. Traditional methods are difficult to effectively identify and classify, especially when used across fields. Word vector representations cannot analyze semantics in combination with context.
The fine-tuned BERT model is used to combine multi-layer Transformer and conditional random field to characterize word vectors, position vectors and segmented vectors, and text features are extracted using multi-head attention mechanism and residual connections, and label transfer features are provided in combination with conditional random field to obtain Entity category.
It improves the comprehensiveness and accuracy of feature extraction of named entity recognition in the field of information security, can better handle complex entities and polysemes, and enhances the semantic understanding of Chinese text and the legality of labeling order.
Smart Images

Figure CN114169329B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to natural language processing in the computer field, and in particular to a named entity recognition method for the information security field. Background Art
[0002] With the increasing number of security risks such as hacker intrusion, Trojan horse propagation, virus infection and DDos service attacks in the Internet environment, if key elements related to information security can be automatically extracted from various attack reports and security analysis texts, it will provide a structured data foundation for building a network security knowledge graph, making the correlation of security threats easier for humans and machines to understand, and providing convenience for dynamic monitoring and early warning of network threats.
[0003] Named entity recognition (NER) involves identifying text fragments belonging to predefined categories within text data and is a major research area in natural language processing. In the information security field, NER is domain-specific and a core step in building an information security knowledge graph. Its primary purpose is to accurately categorize countries, regions, and security-related terminology associated with information security threats.
[0004] Traditional rule-based and dictionary-based named entity recognition methods rely primarily on manually formulated semantics and grammatical and syntactic rules combined with dictionaries. While effective and valuable, most rule-based entity recognition methods consume significant time on rule templates and have limited vocabulary size. Changing rules for changing data is costly, and due to the unique nature of specific domains and dictionaries, rule-based recognition methods are difficult to apply across domains. In machine learning-based named entity recognition, the goal of building a named entity recognition model is to transform entity extraction into a classification problem, achieving efficient recognition through supervised learning. Common machine learning methods, such as hidden Markov models, decision trees, and support vector machines, are used. In recent years, deep learning methods based on neural networks have become a major research direction in named entity recognition. In natural language processing, deep learning can represent and process weighted text semantics through neural network layers, automatically discovering hidden classification elements in raw data through model training. Deep learning methods also possess nonlinear processing capabilities, addressing entity recognition problems across various domains. Deep learning can better capture and learn complex data information.
[0005] As research into entity recognition in the information security field continues to advance, the technical methods of many researchers have gradually matured, leading to continuous improvements in the effectiveness of entity extraction in the information security field. For example, entropy and mutual information metrics can be used to extract threat actions from network threat intelligence, addressing the low accuracy of general extraction methods in the information security field. Alternatively, support vector machines and information extraction systems can be used to extract specific vulnerability and attack concepts from web text related to computer security threats, vulnerabilities, and network attacks. Furthermore, recognition systems based on models such as conditional random fields can be used to extract concepts and descriptions related to network security from text for early automated vulnerability identification and prevention.
[0006] We address some of the challenges of entity recognition in the information security field, including: 1) the complexity and diversity of entity types; 2) the constant emergence of innovative new entities, such as newly developed software or newly established organizations; 3) the mixing of Chinese and English, such as "Tencent QQ"; 4) the phenomenon of abbreviated semantics, such as in "destroying industrial control equipment," where "gong" and "kong" represent "industry" and "control," respectively; and 5) the phenomenon of polysemy, such as "apple," which can represent an unrelated entity in the security field or a company. Furthermore, word vector representations like Word2Vec and Glove use identical vectors for polysemous and polyphonic words, preventing contextual semantic analysis. Summary of the Invention
[0007] In order to solve the above problems, the present invention provides a method for named entity recognition in the field of information security.
[0008] The present invention adopts the following technical solution: a method for identifying named entities in the field of information security, comprising the following steps.
[0009] S100~Use the fine-tuned BERT Chinese model to represent the three dimensions of text data in the security field, namely: word vector, that is, the vector representation of each word itself; position vector, which encodes the position information of the word into a feature vector; 3) segment vector, which is used to distinguish the vector representation of two sentences. Through the separator [SEP] tag, the model inserts the [CLS] tag before the text, and uses the output vector corresponding to this symbol as the semantic representation of the entire text for downstream tasks of the BERT model, and finally obtains the input vector of the model.
[0010] S200~ uses the BERT model to extract feature encoding representations for information security text. Based on a multi-layer Transformer encoder unit stack, a multi-head attention mechanism is used to learn representations of related features in different subspaces. Then, through residual connections and normalization, along with a feedforward neural network, the feature encoding representation of information security text is obtained. The word vector corresponding to each word in the sequence contains the features of other words in the current sentence.
[0011] S300~Use conditional random fields to provide additional label transfer features, obtain transfer scores between label sequences, and select the entity category corresponding to the sequence result with the highest transfer score.
[0012] Step S200 includes taking the following method:
[0013] The input vector is encoded using the Encoder unit of a multi-layer Transformer.
[0014] S201~First, after the vector representation of each word, the input vector of a sentence is formed, that is, , add the word position information PE (Positional Encoding) to get , that is, text embedding representation, where pos represents the position of the word and i represents the dimension of the word:
[0015] ;
[0016] ;
[0017] ;
[0018] S202~Use the multi-head attention mechanism to initialize three weight matrices , ,right Do a linear mapping and get the three dimensions Q, K, and V as Matrix, Q represents the word attributes that match the current word, K represents the attributes of the word itself, and V represents the information contained in the word itself:
[0019] ;
[0020] S203~First find the attention matrix , which represents the relationship between a single word in the text and other words in the sentence, and then uses the score normalization method, that is, using Restore the product result to normal distribution, and the attention mechanism formula is as follows:
[0021] ;
[0022] S204~After that, residual connections are used to avoid the gradient vanishing problem of deep networks:
[0023] ;
[0024] S205~ is normalized to a standard normal distribution to accelerate convergence:
[0025] ;
[0026] S206~After that, add two layers of linear mapping and use activation function to activate , and get the output vector Y:
[0027] .
[0028] Step S300 includes taking the following method:
[0029] For the output vector Y of step S200, we need to get the score of each word corresponding to each label. If there are m possible labels, then there are m n possible annotation score results.
[0030] S301~First, calculate the score of the output vector Y represented by the feature encoding of the text ,in, ,in, Represents each word The corresponding label score distribution, all Sum and define the transfer matrix A ,but is the score of the edge, that is, y i-1 to y i Transfer score:
[0031] ;
[0032] S302~In order to select a better labeling result, after obtaining the score of sequence Y, Softmax is used for normalization to obtain the probability of the labeling result. Represent the exponential of the scores corresponding to all possible sequences:
[0033] ;
[0034] S303~Use the dynamic programming method to reuse the calculation process of the total score of the sequence and train the conditional random field model by maximizing the log-likelihood function:
[0035] ;
[0036] Finally, solve the optimal label sequence based on the score :
[0037] ;
[0038] Obtain the entity label corresponding to each word from the optimal sequence.
[0039] Compared with the existing technology, the present invention obtains multi-dimensional vector features of words, paragraphs and positions in the input vector representation of the text to form a dynamic word vector; in terms of feature extraction, bidirectional encoding is used to better utilize contextual semantics, capture relevant information in different representation subspaces, and extract information from multiple angles to improve the comprehensiveness of feature extraction; it also solves the semantic influence of parts of speech on the text and the legitimacy of the label order of annotated words. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is an overall flow chart of a named entity recognition method for the information security field of the present invention;
[0041] Figure 2 This is a schematic diagram of the framework structure of the named entity recognition model for the information security field;
[0042] Figure 3 Schematic diagram of the embedding layer vector representation structure;
[0043] Figure 4 This is a schematic diagram of the Encoder unit structure. DETAILED DESCRIPTION
[0044] The specific implementation is further described below with reference to the accompanying drawings.
[0045] The present invention applies named entity recognition in the field of information security to identify security entity categories in network security text data. To this end, the present invention proposes an information security entity recognition model that combines BERT and CRF. The model structure diagram is shown in the figure. Figure 2 shown.
[0046] A method for named entity recognition in the field of information security includes the following steps.
[0047] S100~Use the fine-tuned BERT Chinese model to represent the three dimensions of text data in the security field. The input vector of the BERT model is formed by the sum of three different embedding vectors - word vector (Wordpiece TokenEmbedding), which is the vector representation of each word itself; position vector (Position Embedding), which encodes the position information of the word into a feature vector to solve the problem of ignoring the order of words in the attention (Attention) mechanism; segment vector (Segment Embedding), which is used to distinguish the vector representation of two sentences, marked by the [SEP] separator. In addition, the model inserts the [CLS] tag before the text, and uses the output vector corresponding to the symbol as the semantic representation of the entire text for the downstream tasks of the BERT model. The input vector construction structure diagram is shown below. Figure 3 shown.
[0048] S200~After obtaining the input vector of the model, the features of the information security text are extracted through the BERT model. The main structure of the BERT model is composed of multiple layers of Transformer Encoders stacked together. Each layer of BERT consists of an Encoder unit. Its model structure is as follows: Figure 4 As shown in Figure 2, this Transformer structure can use context to capture bidirectional relationships. Encoder is an encoder that converts a variable-length input sequence into a fixed-length vector.
[0049] S201~First, after the vector representation of each word, the input vector of a sentence is formed, that is, , add the word position information PE (Positional Encoding) to get , that is, text embedding representation, where pos represents the position of the word and i represents the dimension of the word:
[0050] ;
[0051] ;
[0052] ;
[0053] S202~Use the multi-head attention mechanism to initialize three weight matrices , ,right Do a linear mapping and get the three dimensions Q, K, and V as Matrix, Q represents the word attributes that match the current word, K represents the attributes of the word itself, and V represents the information contained in the word itself:
[0054] ;
[0055] S203~First find the attention matrix , which represents the relationship between a single word in the text and other words in the sentence, and then uses the score normalization method, that is, using Restore the product result to normal distribution, and the attention mechanism formula is as follows:
[0056] ;
[0057] S204~After that, residual connections are used to avoid the gradient vanishing problem of deep networks:
[0058] ;
[0059] S205~ is normalized to a standard normal distribution to accelerate convergence:
[0060] ;
[0061] S206~After that, add two layers of linear mapping and use activation function to activate , and get the output vector Y:
[0062] ;
[0063] S300~outputs the label sequence through the CRF layer, that is, each word and its corresponding entity label category. In the problem of generating sequence labels, considering that the BERT model has a representation of dependency relationships for labels, in information security entity recognition, the part of speech of Chinese vocabulary has a strong label sequence characteristic. For example, the word "control" is a verb part of speech in "control the destructive power of viruses" and a noun part of speech in "control of the server"; in addition, in terms of the labeling order of words, the label after the B-LOC label should be I-LOC, not I-ORG. Therefore, in the extraction of security elements, the semantic influence of the part of speech on the text and the label order of the labeled words must also be considered. The present invention uses conditional random fields to provide additional label transfer features.
[0064] S301~First, calculate the score of the output vector Y represented by the feature encoding of the text ,in, ,in, Represents each word The corresponding label score distribution, all Sum and define the transfer matrix A ,but is the score of the edge, that is, y i-1 to y iTransfer score:
[0065] ;
[0066] S302~In order to select a better labeling result, after obtaining the score of sequence Y, Softmax is used for normalization to obtain the probability of the labeling result. Represent the exponential of the scores corresponding to all possible sequences:
[0067] ;
[0068] S303~Use the dynamic programming method to reuse the calculation process of the total score of the sequence and train the conditional random field model by maximizing the log-likelihood function:
[0069] ;
[0070] Finally, solve the optimal label sequence based on the score :
[0071] ;
[0072] Obtain the entity label corresponding to each word from the optimal sequence.
Claims
1. A named entity recognition method for information security field, Features: The following steps are included: S100~Use the fine-tuned BERT Chinese model to characterize the three dimensions of security field text data, namely: word vector, that is, the vector representation of each word itself; position vector, which encodes the position information of the word into a feature vector; segment vector, which is used to distinguish the vector representation of two sentences, marked by the separator [SEP]. In addition, the model inserts the [CLS] tag before the text, and uses the output vector corresponding to the tag as the semantic representation of the entire text for the downstream tasks of the BERT model, and finally obtains the input vector of the model; S200~ The BERT model is used to extract the feature encoding representation of information security field text. Based on the stacking of encoder units of multi-layer Transformer, a multi-head attention mechanism is used to learn the expression of different subspace related features. Then, residual connection, normalization and feedforward neural network are used to obtain the feature encoding representation of information security field text. The word vector corresponding to each word in the sequence contains the features of other words in the current sentence. Step S200 includes taking the following method: The input vector is encoded using the encoder unit of the multi-layer Transformer; S201~First, after the vector representation of each word, the input vector of a sentence is formed, that is, , add the word position information PE to get , that is, text embedding representation, where pos represents the position of the word and i represents the dimension of the word: ; ; ; S202~Use the multi-head attention mechanism to initialize three weight matrices , ,right Do a linear mapping and get the three dimensions of Q, K, and V as The matrix, Q represents the word attribute that matches the current word, K represents the attribute of the word itself, and V represents the information contained in the word itself: ; S203~First find the attention matrix , which represents the relationship between a single word in the text and other words in the sentence, and then uses the score normalization method, that is, using Restore the product result to normal distribution, and the attention mechanism formula is as follows: ; S204~Then use residual connections to avoid the gradient vanishing problem of deep networks: ; S205~ is normalized to a standard normal distribution to accelerate convergence: ; S206~After that, add two layers of linear mapping and use activation function to activate , and get the output vector Y: ; S300~Use conditional random fields to provide additional label transfer features, obtain transfer scores between label sequences, and select the entity category corresponding to the sequence result with the highest transfer score.
2. The method for named entity recognition in the field of information security according to claim 1, Features: The step S300 includes taking the following method: For the output vector Y of step S200, we need to get the score of each word corresponding to each label. If there are m possible labels, then possible annotation score results, S301~First, we need to calculate the score of the output vector Y represented by the feature encoding of the text ,in, ,in, Represents each word The corresponding label score distribution, all Sum and define the transfer matrix A ,but is the edge score, that is arrive Transfer score: ; S302: In order to select a better labeling result, after obtaining the score of sequence Y, Softmax is used for normalization to obtain the probability of the labeling result. Represent the exponential of the scores corresponding to all possible sequences: ; S303 ~ The calculation process of the total score of the sequence is reused by the dynamic programming method, and the conditional random field model is trained by maximizing the log-likelihood function: ; Finally, solve the optimal label sequence based on the score : ; The entity label corresponding to each word is obtained from the optimal sequence.
