Security domain named entity recognition method based on pre-training model and Chinese features

Abstract

The invention discloses a security domain named entity recognition method based on a pre-training model and Chinese features, and relates to the technical field of natural language processing, and the method comprises the following steps: 1, injecting domain Chinese vocabularies, and constructing a domain security dictionary; step 2, integrating information in the domain security dictionary into a BERT model to form an SEBERT model; step 3, introducing a residual mechanism on the basis of the KAN network to form an RKAN network; step 4, extracting a multi-scale feature from the SEBERT model by using the RKAN network; and 5, carrying out sequence labeling, capturing a dependency relationship between labels, and identifying the named entity in the security domain.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for identifying named entities in the security field based on a pre-training model and Chinese features. Background Art

[0002] With the rapid development of the industrial Internet, various attacks on enterprises and networks are becoming more frequent, posing a huge threat to the entire society. Building a security knowledge graph to prevent and identify security threats is an important means of current network security, and the key lies in how to accurately identify security incidents from a large amount of unstructured data. Traditional methods rely on the definition of rules by security experts, but this method is not only time-consuming and labor-intensive, but also difficult to guarantee accuracy.

[0003] In the field of network security, security incidents in the industrial Internet environment are unique and complex, involving a large number of proper nouns and emerging terms. Existing NER models based on machine learning, such as Conditional Random Fields (CRF) and Hidden Markov Model (HMM), have solved the problem of rule definition to a certain extent, but lack the ability to understand and recognize these proper words, resulting in the inability to efficiently and accurately identify key entities in the process of building security knowledge graphs. Therefore, developing NER models that can effectively combine domain-specific vocabulary has become the current research focus.

[0004] In addition, deep learning models, such as Bidirectional Encoder Representations from Transformers (BERT) and Bidirectional Long Short-Term Memory (BiLSTM), are also widely used in this field due to their automation and efficiency. However, in the field of network security, due to the lack of specific domain knowledge, these existing models usually ignore vocabulary boundary information when facing Chinese named entity recognition tasks, and have limitations when dealing with tasks involving proprietary vocabulary and complex entity relationships, resulting in the inability to capture the unique vocabulary and complex entity associations in the field of industrial Internet security, resulting in low entity recognition accuracy. Although some technologies have been proposed to optimize the recognition of Chinese vocabulary boundaries, in the field of network security, ordinary vocabulary recognition technology does not significantly improve the model recognition effect.

[0005] Therefore, for named entity recognition in the field of network security, there is an urgent need for a method that can combine domain vocabulary enhancement mechanism and optimize feature representation to improve the performance and accuracy of entity recognition. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the existing model in the field of network security, such as low accuracy in Chinese vocabulary recognition and large errors in understanding domain-specific vocabulary, so as to ultimately improve the overall performance of the model in named entity recognition in the security field.

[0007] To achieve the above object, the present invention provides a security domain named entity recognition method based on a pre-training model and Chinese features, comprising the following steps: Step 1: Inject domain Chinese vocabulary to build a domain security dictionary; Step 2: Integrate the information in the domain security dictionary into the BERT model to form a SEBERT model; Step 3: Introduce the residual mechanism based on the KAN network to form the RKAN network; Step 4: Use the RKAN network to extract multi-scale features from the SEBERT model; Step 5: Perform sequence labeling to capture dependencies between tags and identify named entities in the security field.

[0008] Furthermore, in step 1, an incremental training method is adopted to inject Chinese vocabulary in the said domain.

[0009] Furthermore, the step 1 includes the following sub-steps: Step 1.1, segment the data text to obtain Chinese vocabulary in the field; Step 1.2, inject the Chinese vocabulary of the field into the pre-trained word vector Chinese library, and generate a new professional vocabulary word vector corresponding to the injected Chinese vocabulary of the field while keeping the original word vector in the word vector Chinese library unchanged; Step 1.3: Train the word vector Chinese library so that the word vector Chinese library becomes the domain security dictionary containing professional vocabulary word vectors.

[0010] Furthermore, the step 1.1 uses the jieba word segmenter to segment the data text.

[0011] Furthermore, the step 1.3 uses the Skip-gram model to train the word vector Chinese library.

[0012] Furthermore, the Skip-gram model is a method for identifying surrounding words that can be predicted in a sentence; given a sentence containing Sentences with 1 character The Skip-gram model maximizes the Before and after each character The logarithmic probability of the characters at the positions is used to predict the preceding and following characters of each character. The character with the highest probability in each position is the one that maximizes the average log probability: The Skip-gram model assigns each character in a sentence a unique key from the domain-safe dictionary. Related words of , forming character-word pairs ; in, Represents the number of characters before and after the character to be predicted. Indicates the first characters, and Indicates characters to match the word.

[0013] Furthermore, in step 2, a vocabulary feature fusion mechanism is used to integrate the relevant information of characters and words; the process of the vocabulary feature fusion mechanism is: definition is a character vector, which is the output of a Transformer layer in the BERT model. Representation and The corresponding matching word embedding, Middle The words are represented as follows: in, is a pre-trained word embedding lookup table; Due to the different dimensions of the vectors, the dimensions are aligned again: in, is a dimension of The matrix of is a dimension of The matrix of represents the dimension of word embedding, and Corresponding to the hidden layer size in the BERT model, and is a scalar bias term; Each character matches multiple words, which means that each character has multiple word embedding representations: in, Indicates the number of words matched; Calculate the contribution of different words, determine the final word embedding, and inject weighted vocabulary information into the character vector In the example, vocabulary feature fusion is implemented, and the fused character vector is used as the input of the corresponding Transformer layer in the BERT model to form the SEBERT model.

[0014] Furthermore, the final word embedding is calculated by : in, is the weight matrix of bidirectional attention.

[0015] Furthermore, the weighted vocabulary information is injected into the character vector by the following formula middle: .

[0016] Furthermore, the step 5 uses a CRF model for sequence labeling.

[0017] The security domain named entity recognition method based on the pre-training model and Chinese features provided by the present invention has at least the following technical effects: 1. The technical solution provided by the present invention uses a security dictionary to enhance the BERT model and improve the recognition ability of named entities in the network security domain. Traditional NER models are often difficult to fully understand and recognize complex terms in the field of Industrial Internet Security (IIS). The technical solution of the present invention integrates security vocabulary into a certain Transformer layer of BERT, thereby enhancing the model's ability to understand professional terms and context in the field, and significantly improving the accuracy of entity recognition. Compared with traditional NER methods that use general dictionaries, the method provided by the present invention significantly improves the recognition accuracy of domain-specific vocabulary; 2. On the basis of the SEBERT model, the technical solution provided by the present invention combines the RKAN network and the CRF model to enhance the model's feature extraction and sequence dependency learning capabilities. The RKAN network has a better ability to capture high-dimensional features through the residual mechanism and KAN, optimizes the embedded representation of BERT, effectively solves the complexity of high-dimensional data processing, and improves the ability to distinguish features. In addition, the technical solution provided by the present invention also uses the CRF layer to learn the dependencies between labels, so that the model can better model sequence labels. Experiments show that the SEBERT-RKAN-CRF model outperforms most existing models in identifying entities in the field of industrial Internet security; 3. The technical solution provided by the present invention innovates the architecture and breaks through the traditional BERT-BiLSTM-CRF framework. By replacing BiLSTM with RKAN as the hidden layer, the technical solution provided by the present invention can achieve higher recognition accuracy and F1 score compared to traditional NER models (such as BERT-CRF and BERT-BiLSTM-CRF).

[0018] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of a method model of a preferred embodiment of the present invention; Figure 2 It is a schematic diagram for matching words in the dictionary for specific characters and forming character-word pairs; Figure 3 It is a vocabulary and character feature fusion diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0021] The embodiment of the present invention relates to a named entity recognition (NER) model based on vocabulary enhancement and deep learning, which is mainly used for security incident recognition in the field of Industrial Internet Security (IIS). It uses the pre-training advantage of the BERT model and integrates domain-specific security dictionary knowledge into the BERT model by injecting professional vocabulary in the security field, and proposes a "security dictionary enhanced BERT model", namely the SEBERT model (Security Lexicon Enhanced BERT), and uses the RKAN (Residual-Knowledge-Arnold-Network) network for feature optimization, and uses the conditional random field CRF for sequence labeling. Specifically, by introducing a domain vocabulary enhancement mechanism, a residual optimized feature extraction module, and a sequence labeling capability of a conditional random field, the method provided in the embodiment of the present invention can better identify entities related to security incidents, optimize deep feature extraction, and improve the dependency understanding between labels through sequence labeling. The overall process includes three parts: domain vocabulary enhancement, feature extraction optimization and sequence labeling. The parts are closely linked and work together to effectively solve the problems of inaccurate recognition and insufficient understanding of domain-specific vocabulary in the field of network security of existing NER models, and ultimately improve the overall performance of the model.

[0022] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a security domain named entity recognition method based on a pre-training model and Chinese features, comprising the following steps: Step 1: Inject domain Chinese vocabulary to build a domain security dictionary; Step 2: Integrate the information in the domain security dictionary into the BERT model to form the SEBERT model; Step 3: Introduce the residual mechanism based on the KAN network to form the RKAN network; Step 4: Use the RKAN network to extract multi-scale features from the SEBERT model; Step 5: Perform sequence labeling to capture dependencies between tags and identify named entities in the security field.

[0023] In particular, step 5 uses the CRF model for sequence labeling.

[0024] Example 2 On the basis of Example 1, step 1 adopts the incremental training method to inject Chinese vocabulary into the field. As a language with rich connotations and vocabulary, training the proprietary word vectors of all words from scratch is an extremely time-consuming and labor-intensive task. Therefore, the embodiment of the present invention chooses to train on the basis of the currently published pre-trained word vector Chinese library. After loading the pre-trained word vectors, the embodiment of the present invention searches the word vector library and finds that most of the professional vocabulary for industrial control safety does not exist in the word vector. Therefore, the data in the field of industrial control safety is added to the word vector database, and the method adopted is the incremental training method.

[0025] In particular, step 1 includes the following sub-steps: Step 1.1: Segment the data text to obtain domain Chinese vocabulary; specifically, this step uses the Jieba word segmenter to segment the existing data text; Step 1.2, inject the domain Chinese vocabulary into the pre-trained word vector Chinese library, and generate new professional vocabulary word vectors corresponding to the injected domain Chinese vocabulary while keeping the original word vectors in the word vector Chinese library unchanged; Step 1.3: Train the Chinese word vector database so that the Chinese word vector database becomes a domain security dictionary containing professional vocabulary word vectors.

[0026] Example 3 On the basis of Example 2, step 1.3 uses the Skip-gram model to train the word vector Chinese library, so that the word vectors in the word vector Chinese library are more suitable for the field.

[0027] Specifically, the Skip-gram model is a method for identifying words that can predict surrounding words in a sentence; given a sentence containing Sentences with 1 character The Skip-gram model maximizes the Before and after each character The logarithmic probability of the characters at the positions is used to predict the preceding and following characters of each character. The character with the highest probability in each position is the one that maximizes the average log probability: After the training is completed, the embodiment of the present invention has a dictionary that includes professional vocabulary word vectors based on the original word vectors.

[0028] Given a dictionary containing specialized vocabulary and by Sentences consisting of characters , the Skip-gram model assigns each character in a sentence a Related words of , forming character-word pairs , where represents the number of characters before and after the character to be predicted, represents the th character of the sentence, and represents the word matched by the th character. As Figure 2 shown, when the sentence character group is [Shock, Network, Disease, Virus, Invasion], the embodiment of the present invention uses the consistency of characters to select the most relevant vocabulary corresponding to each character from the dictionary.

[0029] Example 4 Based on Example 3, in step 2, a lexical feature fusion mechanism is used to integrate the relevant information of characters and words.

[0030] As Figure 3 shown, the process of the lexical feature fusion mechanism is as follows: Define as a character vector, which is the output of a certain Transformer layer in the BERT model, represents the embedding of the matching word corresponding to , and the rd word in is represented by the following method: where is a pre-trained word embedding lookup table; Due to the different vector dimensions, the dimensions are aligned again: where is a matrix with a dimension of , is a matrix with a dimension of , represents the dimension of the word embedding, and corresponds to the hidden layer size in the BERT model, and are scalar bias terms; Each character matches multiple words, which means that each character has multiple word embedding representations: where represents the number of words matched; Calculate the contribution values of different words, determine the final word embedding, and inject the weighted lexical information into the character vector In the BERT model, we implement vocabulary feature fusion and use the fused character vector as the input of the corresponding Transformer layer in the BERT model to form the SEBERT model.

[0031] In particular, since it is not possible to input every word vector, different words contribute differently in different contexts. The final word embedding is calculated by : in, is the weight matrix of bidirectional attention.

[0032] Specifically, weighted vocabulary information is injected into the character vector by the following formula middle: .

[0033] Finally, the fused character vector is used instead of simple character embedding as the input of a specific Transformer layer in the BERT model to form the SEBERT model.

[0034] In particular, in order to effectively process high-dimensional data, an embodiment of the present invention adopts a KAN network to replace an ordinary fully connected layer to decompose complex high-dimensional functions into a combination of simple one-dimensional functions, that is, by changing the fully connected layer to a KAN layer in the last layer of the model and making appropriate modifications to the parameters.

[0035] At the same time, the KAN network is modified. Based on the KAN network, a residual mechanism is introduced to form the RKAN network, which enables the model to learn multi-scale features, improve the overall performance and promote better gradient propagation.

[0036] Finally, in step 5, the CRF model is used to capture the dependencies between labels in the sequence data. For example, in the BIO annotation strategy, the "BX" label always comes before the "IX" label, and the CRF model can take this dependency into account. Usually in the named entity recognition task, adding a CRF model can effectively improve performance and indicators, which is also determined by the characteristics of the named entity recognition task and the application scope of the CRF model.

[0037] The technical solution provided by the present invention has strong adaptability to domain-specific data sets and is applicable to a variety of practical scenarios. The embodiment of the present invention greatly improves the recognition ability of specific named entities in the field of industrial Internet security by incrementally training the special dictionary in the field of industrial Internet security and combining it with the existing pre-trained embedded dictionary. Especially in data sets (such as CDTier data sets), the model can effectively identify entities including attackers, regions, tools, industries, and activities. This feature makes the present invention not only suitable for traditional encrypted communications, but also can be widely used in multiple fields such as threat intelligence analysis and the construction of industrial security knowledge graphs, and has extremely high practical value and application prospects.

[0038] In addition, the technical solution provided by the present invention has wide applicability to Chinese and English entity recognition, and takes into account the universality and domain specificity of the model. The SEBERT-RKAN-CRF model provided not only performs well in the field of industrial Internet security, but can also be applied to named entity recognition tasks in other fields by adjusting the dictionary and pre-training model. By introducing domain-specific vocabulary information, the present invention significantly enhances the model's ability to capture complex named entities, especially in Chinese NER tasks, effectively solving the problem of fuzzy vocabulary boundaries.

[0039] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A security domain named entity recognition method based on a pre-trained model and Chinese features, characterized in that: The method comprises the following steps: Step 1: Inject domain Chinese vocabulary to build a domain security dictionary; Step 2: Integrate the information in the domain security dictionary into the BERT model to form a SEBERT model; Step 3: Introduce the residual mechanism based on the KAN network to form the RKAN network; Step 4: Use the RKAN network to extract multi-scale features from the SEBERT model; Step 5: Perform sequence labeling to capture dependencies between tags and identify named entities in the security field.

2. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 1, characterized in that: Step 1 uses an incremental training method to inject Chinese vocabulary in the field.

3. The security domain named entity recognition method based on pre-training model and Chinese features as claimed in claim 2, characterized in that: The step 1 comprises the following sub-steps: Step 1.1, segment the data text to obtain Chinese vocabulary in the field; Step 1.2, inject the Chinese vocabulary of the field into the pre-trained word vector Chinese library, and generate a new professional vocabulary word vector corresponding to the injected Chinese vocabulary of the field while keeping the original word vector in the word vector Chinese library unchanged; Step 1.3: Train the word vector Chinese library so that the word vector Chinese library becomes the domain security dictionary containing professional vocabulary word vectors.

4. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 3, characterized in that: The step 1.1 uses the jieba word segmenter to segment the data text.

5. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 3, characterized in that: The step 1.3 uses the Skip-gram model to train the word vector Chinese library.

6. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 5, characterized in that: The Skip-gram model is a method for identifying words that can predict surrounding words in a sentence; given a sentence containing Sentences with 1 character The Skip-gram model maximizes the Before and after each character The logarithmic probability of the characters at the positions is used to predict the preceding and following characters of each character. The character with the highest probability in each position is the one that maximizes the average log probability: The Skip-gram model assigns each character in a sentence a unique key from the domain-safe dictionary. Related words of , forming character-word pairs ; in, Represents the number of characters before and after the character to be predicted. Indicates the first characters, and Indicates characters to match the word.

7. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 6, characterized in that: In step 2, a vocabulary feature fusion mechanism is used to integrate the relevant information of characters and words; the process of the vocabulary feature fusion mechanism is: definition is a character vector, which is the output of a Transformer layer in the BERT model. Representation and The corresponding matching word embedding, Middle The words are represented as follows: in, is a pre-trained word embedding lookup table; Due to the different dimensions of the vectors, the dimensions are aligned again: in, is a dimension of The matrix of is a dimension of The matrix of represents the dimension of word embedding, and Corresponding to the hidden layer size in the BERT model, and is a scalar bias term; Each character matches multiple words, which means that each character has multiple word embedding representations: in, Indicates the number of words matched; Calculate the contribution of different words, determine the final word embedding, and inject weighted vocabulary information into the character vector In the example, vocabulary feature fusion is implemented, and the fused character vector is used as the input of the corresponding Transformer layer in the BERT model to form the SEBERT model.

8. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 7, characterized in that: The final word embedding is calculated as follows : in, is the weight matrix of bidirectional attention.

9. The security domain named entity recognition method based on a pre-training model and Chinese features as claimed in claim 8, characterized in that: The weighted vocabulary information is injected into the character vector by the following formula middle: 。 10. The security domain named entity recognition method based on pre-training model and Chinese features as claimed in claim 1, characterized in that: The step 5 uses the CRF model to perform sequence labeling.

Images